A Marketplace Is Not a Test Range

The standards are broader than the sales pitch

The new counter-UAS evaluation material contains a useful challenge to the industry's most compressed promises. It does not ask only whether a system can detect or defeat a drone. Readable sections address operator workload, interfaces, integration, reliability, cybersecurity, and lifecycle costs. Its discussion of mission impact asks whether a system contributes a net benefit while accounting for the risks it creates. That is a more serious conception of capability than a spectacular successful engagement taken in isolation.¹

JIATF-401 announced publication of the full unclassified material in late April, following the March release of standardized guidelines. The stated purpose is to improve testing, data generation, and joint evaluation. The publication matters because a buyer can now ask a supplier questions in a language that other public institutions may also recognize. Common expectations can reduce the waste created when every interested organization requests a different demonstration without retaining comparable information.¹

The standardization effort should therefore be judged on what it actually contains. It would be inaccurate to criticize it as though operator burden, integration, or risk had simply been forgotten. The difficult question is how those dimensions will be evaluated together, how the resulting judgments will be used, and which conclusions can properly travel from one setting to another. A list of dimensions is the beginning of that work, not the completed comparison.

The title's distinction between a marketplace and a test range is not an argument that either institution is unnecessary. A marketplace helps buyers transact. A range creates conditions in which claims can be examined. A useful procurement system connects the two without pretending their functions are identical. The listing needs the reasons generated by evaluation; the evaluation needs a route through which its findings can affect actual purchases. Speed is valuable when that connection works and misleading when it merely makes a weak claim easier to buy.

There is an economic consequence to choosing what gets measured. Suppliers invest in the activities that help them gain customers. A market that rewards only an impressive visible effect will encourage a different product from one that also rewards low operator burden, straightforward integration, maintainability, and candid reporting of limits. Evaluation criteria can therefore shape industrial development before a particular purchase is made. They are part of the market's design, not neutral paperwork applied after the important decisions.

This makes their administration a public responsibility. A standard can open competition by making expectations clear and evidence reusable. It can also exclude capable smaller firms if the cost of demonstrating compliance is needlessly high. The challenge is to obtain information that changes decisions while avoiding elaborate demands that merely reward the supplier with the largest proposal team. Rigor should be visible in the quality of a comparison, not measured by the weight of the submission.

As of May 8, the opportunity is to make that ambition practical. A common evaluation language now exists in public form. Its value will depend on how evaluators interpret it, how manufacturers respond, and whether buyers can understand the boundaries of the result. The most important future document may be neither the catalogue nor the standard itself, but a decision explaining why particular evidence was sufficient for a particular use.

Comparison without a universal winner

Standardization has an attractive promise: different products can be measured against a common problem. That promise becomes dangerous if it is translated into a belief that every purchaser should receive the same answer. A mobile system serving a temporary requirement and an installation supporting continuous activity may place different demands on personnel, support, and integration. A shared vocabulary should help describe those differences rather than erase them in pursuit of one tidy ranking.

The first discipline is to state the task before comparing products. A buyer needs to know which function the equipment is meant to provide and which other parts of the system will supply the remainder. A product that performs one function well should not inherit credit for an entire defensive arrangement. Nor should a component be judged as a failed complete solution when it was never offered or tested as one. Clear boundaries make both criticism and comparison fairer.

The second discipline is to distinguish what was observed from what was inferred. An evaluation can directly measure a result under defined conditions, while a procurement decision may need an estimate about longer service or a different location. Inference is unavoidable. It becomes more defensible when the decision identifies the difference and explains why the available evidence is relevant. The problem is not that officials must exercise judgment. It is that a numerical presentation can conceal the judgment while appearing to eliminate it.

Weighting is part of that judgment. A system might offer better technical performance but require a larger support organization. Another might be easier to integrate but less suited to a demanding part of the task. The institution should be able to explain how these tradeoffs relate to its mission. A common evaluation record can support more than one reasonable purchase, and disagreement can reveal a difference in requirements rather than a defect in the data.

There is a related danger of optimizing for the examination. Once a demonstration becomes the principal route to a contract, a supplier has a rational incentive to devote scarce development effort to the conditions it expects to encounter there. Some of that effort is beneficial: clear expectations can improve products. The institution should still ask whether the resulting improvement addresses the user's actual difficulty or merely raises a score. This is a reason to combine common measures with independent user feedback and to examine ordinary support work alongside the most visible event. A system should earn credit for becoming more useful, not simply for becoming more familiar with the government's preferred presentation of usefulness.

This also protects suppliers from misleading comparisons. If a public evaluation rejects a product for one mission, another buyer should be able to understand why before treating the result as a general verdict. The same principle applies to success. A favorable result should identify what it established, leaving a later buyer able to judge relevance. Neither reputation nor a numerical score should become an all-purpose certificate detached from the conditions that gave it meaning.

Cost requires similar care. The published material includes procurement, installation, maintenance, and personnel costs as well as engagement-related measures. Those categories acknowledge that a cheap visible action can depend on an expensive supporting arrangement. A buyer needs a realistic period of use and a clear account of what is included before comparing alternatives. Otherwise the least expensive part of the transaction can dominate a discussion whose actual financial burden appears elsewhere.¹

The result should be a common basis for explanation, not a universal answer. That is a more modest promise than the dream of automatic selection, but a more useful one. It permits accountability when a decision is challenged and correction when requirements change. It also helps a buyer recognize the point at which an apparently comparable offering is asking the institution to supply a capability that another vendor included in its price.

A test should be able to change the decision

The public value of evaluation is clearest when it can produce an inconvenient result. If every demonstration must confirm the wisdom of a purchase already celebrated, testing becomes a ceremony. A useful assessment can narrow a proposed use, require a correction, reveal an additional support burden, or support a decision not to proceed. Those outcomes protect the value of public investment even when they are less attractive to the people who arranged the event.

This requires clarity about the result being sought. A supplier and evaluator should understand what counts as an attempt, an incomplete observation, a failure, and an inconclusive result. Otherwise a favorable percentage can depend as much on accounting choices as on the equipment. A comparison becomes particularly weak when one product's difficult cases disappear from the record while another's remain visible. Retaining the full relevant sequence makes later scrutiny possible without demanding that every test produce a perfect answer.

The purpose is not to publish a tactical recipe or sensitive operational limits. It is to preserve enough information for authorized buyers and reviewers to understand the claim. Public summaries can identify the kind of evaluation, the existence of unresolved limitations, and the scope of the decision without disclosing details that would undermine protection. Confidentiality and intelligibility are compatible when the institution decides deliberately what each audience needs to know.

Repeatability should also be understood realistically. A field setting cannot be made identical in every respect, and a product may change during a development program. Evaluators should preserve the important conditions and record meaningful differences. That allows a later team to distinguish an improvement from a more favorable test, or a new weakness from a changed requirement. Without that history, repeated demonstrations may create a growing collection of impressive results that cannot actually be compared.

The supplier's presence deserves attention. A prototype may depend on engineers who understand its unusual behavior and can restore it quickly. Their participation is often appropriate during development. It should be recorded when the eventual customer expects ordinary personnel to operate and maintain the system. The question is what organization was required to obtain the result, and whether that organization is included in the proposed delivery.

An evaluation should also identify the decision it is intended to support. An exploratory event can reveal whether an approach merits more development; it need not carry the burden of final operational acceptance. A later assessment may examine whether the intended user can sustain the delivered arrangement. Confusing those purposes can produce both excessive demands and unjustified confidence. A promising early result may be rejected because it cannot yet answer every question, while a modest exploratory success may be promoted as if the remaining work were complete. Naming the stage gives the result a fair meaning and makes it easier to decide what evidence is needed next.

Unfavorable findings can become a shared asset if institutions have a way to use them responsibly. A problem with a configuration should inform the people considering it, and a successful correction should be linked to the problem it addressed. A system that preserves only favorable outcomes creates repeated rediscovery at public expense. A system that preserves failures without their context can unfairly stigmatize products that have improved. The useful record contains the relationship between observation, change, and subsequent evaluation.

Standards help by making those relationships less dependent on personal memory. They cannot eliminate the pressure to declare success quickly. That is a question of incentives and leadership. A program that rewards timely discovery of a serious limitation is better positioned to improve than one in which the discovery threatens everyone's standing. The institution should make it possible for testing to change its mind before an operational event changes it at a higher cost.

A new version meets its users

The April 29 government account of Bumblebee V2 provides a concrete setting for these principles. It describes an April 22 event at Fort Bragg involving twenty paratroopers and prototypes purchased earlier in the year. The article calls it the first operational test of those prototypes and reports feedback from users. This is valuable evidence of a development and training activity. It does not contain a public dataset demonstrating performance across every condition associated with the system's intended use.²

The wording inside the article is itself instructive. Officials discuss automated target recognition and interception, while captions describe the system as under development to add autonomous capabilities. The responsible interpretation preserves both the stated ambition and the prototype status. It should not silently turn every forward-looking description into a mature capability available identically in all delivered units. Development language is particularly important when the feature being promoted changes the relationship between operator and machine.²

User participation can reveal the part of capability that specifications miss. A system may require people to interpret several kinds of information, coordinate with others, and recognize when the available evidence is inadequate. A more capable automated feature does not remove the need to understand that relationship. It can change where the user's attention is needed and which failures become difficult to notice. Evaluation should follow the actual allocation of work rather than assume that more automation always means less burden.

Twenty participants are a meaningful group for practical feedback, but the number alone does not establish how representative their experience is. The public account does not supply the full distribution of prior skill or every task attempted. That does not diminish the event. It defines the kinds of conclusion the event can support publicly. Early user work is useful precisely because it helps identify what should be examined next, before the product is treated as a finished answer.

The transition between versions also needs a clear record. Evidence about an earlier system can inform the new version without automatically validating it. Changed cameras, sensors, or software may improve performance while altering dependencies and workload. A receiving unit should know what changed, which observations remain relevant, and which claims require new support. Otherwise a familiar product name can carry confidence forward faster than the underlying evidence.

Supply changes belong in that account as well. A manufacturer may need to substitute an unavailable input to maintain delivery. The replacement can be entirely reasonable and still require a question about which earlier observations remain applicable. Test records should identify the relevant configuration well enough to make that question answerable. The objective is not to penalize a firm for keeping production moving. It is to avoid transferring confidence silently between objects that differ in a way the evaluator considered important. That discipline makes domestic production more credible because the resulting product can be connected to a known technical and support history, rather than relying on origin as its only recognizable feature.

Procurement should make room for this developmental reality. Buying prototypes can be a rational way to learn, especially when the alternative is waiting for a requirement to become obsolete. The purchase should be described and managed as such. The customer needs a route for accepting changes, reporting difficulties, and deciding what constitutes readiness for the next stage. A prototype becomes more valuable when the institution learns from it; it becomes more dangerous when its purchase is taken as an obligation to defend every claim made for it.

The strongest outcome from the Fort Bragg work would therefore be a set of decisions that improves the product and the receiving organization together. That might include changed training, a revised interface, or a narrower understanding of appropriate use. The public need not see every detail to understand that progress is occurring. It should be able to distinguish the existence of hands-on work from a generalized claim that the product has completed its development.

Five installations, more than one environment

The May 6 directed-energy pilot announcement raises the same issue at a larger institutional scale. JIATF-401 selected Fort Huachuca, Fort Bliss, Naval Base Kitsap, Grand Forks Air Force Base, and Whiteman Air Force Base for a program intended to examine capabilities across different environments and missions. The announcement says deployment plans will be finalized during the following 180 days, enabling operations later in the year. Selection is therefore a program milestone, with substantial implementation still prospective at this dateline.³

The diversity of sites is a useful feature if it becomes an opportunity for comparison. A pilot should reveal what must be supplied locally and which assumptions survive a change of setting. The existence of five locations does not by itself produce that knowledge. The program needs a way to retain important differences and relate each outcome to the conditions under which it occurred. Otherwise a multisite pilot can become five disconnected demonstrations presented as one national conclusion.

The announcement places lasers and high-powered microwave systems within a broader layered approach. It also refers to earlier cooperation with the FAA and safety assessment. These are consequential institutional claims. They should be kept distinct from a public technical certification covering every future configuration and location. The detailed operating basis is not reproduced in the announcement, and this essay does not invent it. Continued coordination is part of the work, rather than an admission that the work has failed.³

A local commander needs more than confidence in the principal equipment. The installation must support it, the responsible organizations must understand their duties, and its operation must fit other legitimate activity. This is why the standards' attention to mission impact and risk is important. A defensive system should be assessed for its whole contribution, including burdens and effects outside the immediate task. Technical success alone can leave an unsuitable arrangement if those surrounding obligations have been ignored.

The pilot can also produce knowledge about sustainment. A technology may perform well during a carefully supported event while requiring a maintenance organization that the installation does not yet possess. The program should identify that requirement early enough to finance and train for it. Treating support as an afterthought can turn an impressive delivery into equipment used less reliably than the original proposal assumed. A realistic account should follow the system beyond introduction into ordinary service.

Common reporting would help different installations learn from each other without pretending their experience is identical. A fault corrected at one location might matter elsewhere; a local difficulty might be irrelevant to another mission. The institution needs both the shared record and the context that prevents careless generalization. That is one of the principal advantages a public pilot can offer over separate private demonstrations: the government can retain the comparative knowledge for use across future decisions.

The public should judge the pilot by that contribution as well as by visible fielding. If it discovers a limitation, the result can still be valuable. If it confirms a useful capability, the result should identify the conditions that made it useful. A program that only counts selected sites will have measured administrative progress. A program that explains what changed in its understanding will have created an asset that can improve the next purchase.

The market needs an institution that can learn

The surrounding civil and industrial record gives this argument a wider purpose. The FAA's April 25 Drone Safety Day campaign addresses responsible operation in shared airspace. It is outreach, not a counter-drone product evaluation or evidence of a measured reduction in incidents. Its relevance is the reminder that legitimate drone use remains part of the environment defensive systems must inhabit. A market organized entirely around threatening targets would neglect the public activity it is also supposed to protect.⁴

Industrial ambition adds another pressure. Reuters' April 29 account of Ukraine's diplomatic partnerships describes the challenge of translating interest into delivery, while preserving the dependence of some capabilities on skilled people and coordinated practice. The story is a reminder that demand can be established faster than support and production can be organized. It does not provide an independently audited capacity schedule for every firm. A buyer needs that practical detail before treating a partnership as an available capability.⁵

These pressures make a common evaluation system valuable. Manufacturers need expectations stable enough to guide development. Buyers need findings usable enough to inform selection. Operators need an arrangement that reflects their work. The public needs some assurance that rapid spending is producing more than a sequence of endorsements. The standards can help align these interests, provided their application remains open to correction and the evidence stays connected to what is actually purchased.

Access to evaluation should not become a closed privilege. A new supplier may have a useful approach but limited resources to reproduce several expensive demonstrations. Reusable results, clear entry requirements, and proportionate follow-up can lower that burden. At the same time, an institution must retain the ability to reconsider an incumbent's status when a product changes or fails to meet its commitments. Fair competition requires a route in and a meaningful possibility of reassessment.

There is an obligation on buyers as well. They should not demand a broad claim and then blame the vendor when the claim proves too vague to enforce. Requirements need to identify the desired result, the support expected, and the conditions that matter. A purchasing organization that cannot explain its own task will struggle to distinguish a strong product from a strong presentation. Better standards should improve the buyer's questions as much as the seller's evidence.

The public return is an institution that remembers. It should know why a product was selected, which limitations remained, what changed after testing, and how the delivered configuration relates to the one examined. That memory permits faster decisions later because relevant work can be reused intelligently. Without it, procurement speed merely delivers the same uncertainty to more customers, each of whom must rediscover the limits at its own expense.

Keeping that memory requires assigned responsibility. Someone must maintain the evaluation record when personnel rotate, decide how a supplier's correction relates to an earlier finding, and ensure the buyer sees the current account. These tasks are less visible than organizing a demonstration, but they preserve its value. A standard that assumes this work happens automatically will gradually lose contact with the products it helped select. Funding the continuing administration of evidence is therefore part of funding evaluation, rather than overhead that can be removed once the event has been photographed.

As of May 8, publication of the criteria, prototype work, and pilot selection establish useful beginnings. Their success will depend on the decisions they improve. A marketplace can make a capability easier to acquire; a test range can make a claim easier to judge. The durable achievement is to join those functions in a public system able to retain context, recognize failure, and change course. That is how a rapidly growing counter-drone market can become more trustworthy as it becomes faster.

Notes and sources