On Counterfeit Chips in US Military Hardware

Amendment 1092 to the Defense Authorization Act of 2012 is a well-intentioned but misguided provision outlining measures designed to reduce the prevalance of counterfeit chips in the US military supply chain.

The Defense Authorization Act already has drawn flack for a provision that gives the US military authorization to detain US citizens indefinitely without trial, and I think it rather ironically requires an assessment of the US Federal Debt owed China as a potential “National Security Risk” (section 1225 of HR1540) — anyone want to take bets as to whether the conclusion of this assessment leads to prioritizing deficit reduction as a national security issue, or if it leads to justifying further borrowing from China to build up a military to fend off its biggest creditor?

Under the proposed anti-counterfeit amendment, first-time offenders can receive a $5 million fine and 20 years prison for individuals, or $15 million for corporations; a penalty comparable to that of trafficking cocaine. While the amendment explicitly defines “counterfeit” to include refurbished parts represented as new, the wording is regrettably vague on whether you must be willfully trafficking such goods to also be liable for such a stiff penalty.

If you took a dirty but legitimately minted coin and washed it so that it looked mint condition and then sold it to a collector as mint quality, nobody would accuse you of counterfeiting. Yet, this amendment puts a 20 year, $5 million penalty on not only the act of counterfeiting chips destined for military use, but potentially the unwitting distribution of such chips that you putatively bought as new but couldn’t tell yourself if they were refurbished. Unfortunately, in many cases an electronic part can be used for years with no sign of external wear.

The amendment also has a provision to create an “inspection program”:

(b) Inspection of Imported Electronic Parts –

(1) … the Secretary of Homeland Security shall establish a program of enhanced inspection by U.S. Customs and Border patrol of electronic parts imported from any country that has been determined by the Secretary of Defense to have been a significant source of counterfeit electronic parts …

It’s one thing to inspect fruits and vegetables as they enter the country for pests and other problems; but it is misguided to require Customs officers to become experts in detecting fakes, and/or to burden vendors with the onus of determining whether parts are authentic, particularly with such high penalties involved and the relative ease that forgers can create high-quality counterfeit parts.

To better understand the magnitude of the counterfeiting problem, it’s helpful to know fakes are made. The fakes I’ve seen fall into the following broad categories:

1) Trivial external mimicry. Typically these are empty plastic packages with authentic-looking topmarks, or remarked parts that share only physical traits with the authentic parts (for example, a TTL logic chip in an SO-20 case remarked as an expensive microcontroller that uses the same SO-20 case). I consider this technique trivial because it is so easy to detect during factory test; in the worst case you are sold a thin mixture of authentic and conterfeit parts so that testing just one part out of a tube or reel isn’t good enough. However, in all cases the problem is discovered before the product ships so long as the product overall is thoroughly tested.

2) Refurbished parts. These are authentic parts recovered from e-waste that have been desoldered and reprocessed to appear as new. These are very difficult to spot since the chip is in fact authentic, and a skilled refurbisher can create stunningly authentic-looking results that can only be discriminated with the use of electronic micsoscopes and elemental/isotopic analysis. I also include in this category parts that are new only the sense they have never been soldered onto a board, but were stored improperly (for example, in a humid environment) and should be scrapped, but were subsequently reconditioned and sold like new.

3) Rebinned parts. These are parts that were authentic, and perhaps have never been used (so can be classified as “new”), but have their markings changed to reflect a higher specification of an identical function. A classic example is grinding and remarking CPUs with a higher speed grade, or more trivially parts that contain lead marked as RoHS-compliant. However, it can get as sophisticated as vendors reverse engineering and reprogramming the fuse codes inside the chip so that the chip’s electronic records match the faked markings on top; or vendors have been known to do deep hacks on Flash drive firmware so that a small memory can appear to a host OS as a much larger memory, going so far as to “loop” memory so that writes beyond the capacity of the device appear to succeed.

4) Ghost-shift parts. These are parts that are created on the exact same fabrication facility as authentic parts, but run by employees without authorization of the manufacturer and never logged on the books. Often times they are assigned a lot code identical to a legitimate run, but certain testing steps are skipped. These fakes can be extremely hard to detect. It’s like an employee in a mint striking extra coins after-hours.

5) Factory scrap. Factory rejects and pilot runs can be recovered from the scrap heap for a small bribe, and given authentic markings and resold as new. In order to avoid detection, workers often replace the salvaged scrap with physically identical dummy packages, thus foiling attempts to audit the scrap trail.

6) Second-sourcing gone bad. Second-sourcing is a standard industry practice where competitors create pin-compatible replacements for popular products in order to create price competition and strengthen the supply chain against events like natural disasters. The practice goes bad when inferior parts are re-marked with the logos of premium brands. High-value but functionally simple discrete analog chips such as power regulators are particularly vulnerable to this problem. Premium US brands can command a 10x markup over Asian brands, as “drop-in replacement” Asian-brand parts are notorious for spotty quality, cut corners and poor parametric performance. However, there is a lot of money to be made buying blanks from the second source fab and remarking them with authentic-looking top marks of premium US brands. In some cases there are no inexpensive or fast tests to detect these fakes, short of decapsulating the chip and comparing mask patterns and cross-sections.

In the case of the US Military, they have a unique problem where they are one of the biggest and wealthiest buyers of really old parts. Military designs have shelf lives of decades, but parts have production cycles of only years. It’s like asking someone to build a NeXT Cube motherboard today using only certifiably new parts; no secondhand or refurbished parts allowed. I don’t think it’s possible.

The impossibility of this situation may force military contractors to be complicit in the consumption of counterfeit parts. For example, the fake parts in the P-8 Poseidon were “badly refurbished”. A poor refurbishing job is probably detectable with a simple visual inspection, so even though people are quick to point fingers at China, maybe part of the problem is that the contractor was lax in checking incoming stock — or perhaps looking the other way, because if these are the last parts of its kind in the world, what else can they do?

Another part of the senate hearings revealed that L3 bought counterfeit video memory chips destined for C-27J aircraft from Global IC Trading Group. Well … duh. Global IC ain’t Digikey … they specialize in trading excess, overruns and secondhand goods. The prices are often good, but I only go to them if I’m really in a bind, and I’m willing to accept odd lots to get production moving at any cost. L3′s big enough to have a professional sourcing group aware of that, and thus exercise extreme caution when buying from such vendors.

My guess is that the stocks of any distributor in the secondhand electronics business are already flooded with undetected counterfeits. Remember, only the bad fakes are ever caught, and chip packaging was not designed with anti-counterfeiting measures in mind. While all gray market parts are suspect, that’s not necessarily a bad thing. Gray markets play an essential role in the electronics ecosystem; using them is a calculated but sometimes unavoidable risk.

While the situation is clearly a mess now, some simple measures going forward could help fix things for the future. One could involve embedding anti-counterfeit measures in chips approved for military use. For chips larger than 1cm, a unique 2-D barcode can be applied with laser markings. The equipment to do such laser-marking is relatively commonplace today in chip packaging facilities. The efficacy of such techniques has been proven in biotech, where systems such as Matrix 2D are deployed to track disposable sample tubes in biology labs. Despite a tiny footprint, the codes are backed with a guarantee of 100% uniqueness. Another potential solution is to mix a UV dye into the component’s epoxy that changes fluorescence properties upon exposure to reflow temperatures. If the dye is distributed through the plastic body of the case, the change will be impossible to remove with grinding alone.

A second partial measure could be to manage e-waste better. E-waste is harvested in bulk for used parts. One can purchase crudely desoldered MSM7000-series chips (the brains of many Android smartphones) by the pound, at around ten cents for a chip. These chips are then cleaned up, reballed and sometimes remarked, put into tapes and reels and sold as brand new, commanding over a 10x markup. Thus, a single batch of chips can net thousands of dollars, making it a compelling source of income for skilled labor that would otherwise work in a factory for $200 per month.

If we stopped shipping our e-waste overseas for disposal, or at least ground up the parts before shipping them over, then the feedstock for such markets would be reduced. It could also create jobs domestically for processing the e-waste, which by the way is a source of gold comparable to the richest gold ore. On the other hand, I’m of the opinion that in the big picture this sort of component-level recycling is actually quite good for the environment and the human ecosystem. Upon disposal, most electronics still have years of serviceable life in them, and there is a hungry market for technology in emerging economies that cannot be met with new parts purchased on the primary market.

A final option could be to establish a strategic reserve of parts. A production run of military planes is limited to perhaps hundreds of units, and so I imagine the lifetime demand of a part including replacements is limited to tens of thousands of units. I can fit ten thousand chips in the volume of a large shoebox; at least physically, it’s not an unmanageable proposition. These are small volumes compared to consumer electronics volumes. I imagine that purchasing a reserve of raw replacement components for critical avionics systems would only add a fraction of a percent to the cost of an airplane, and could even lead to long term cost savings as manufacturers can achieve greater scale efficiency if they run one large batch all at once. This could be a foolproof method to ensure supply trustability for critical military hardware.

26 Responses to “On Counterfeit Chips in US Military Hardware”

  1. [...] Bunnie had a fantastic overview of some of the issues with Amendment 1092 to the Defense Authorization Act of 2012… Amendment 1092 to the Defense Authorization Act of 2012 is a well-intentioned but misguided provision outlining measures designed to reduce the prevalence of counterfeit chips in the US military supply chain. [...]

  2. foo says:

    Could the open hardware movement one day play a role here? With open designs one can do multiple production runs, avoiding the need for a strategic reserve. It would be great if you could blog about open hardware and computing security.

  3. Roland Dobbins says:

    The other alternative is to realize the folly of purchasing military components from one’s largest geostrategic competitor, and instead produce them in one’s own country while employing one’s own fellow citizens to make them – but in the race to maximize defense contractor profits, that will never happen.

    • bunnie says:

      I suspect many of these “counterfeit” chips were actually originally manufactured in the US, and then exported overseas. The chips were used, then extracted from devices, refurbished, and sold back to the US and represented as brand new US stock.

      So the issue isn’t strictly a matter of provenance; it’s a matter of the total life cycle of a component in a global market.

      • This is one of those moments I wish there was a “Like” button here.

        I wish I could find the link again, but I remember reading a couple of articles talking about some of the most common ways that counterfeits get into the supply chain. A huge source is actually US made material, that either becomes electronic waste, that is then salvaged & harvested.. or as you say, things that didn’t get stored well (or for what ever reason) “failed” and tossed.

        You have a great description of how that then re-enters the supply chain. One of the things I’m hoping that comes around this amendment is also updating the CBP redacting process. Yes, it puts burden back on the original manufacturer to ID bar codes and lot numbers… but it is one more step in catching the counterfeits as they re-enter the states and the supply chain.

        Great post! Thank you :)

  4. [...] Bunnie had a fantastic overview of some of the issues with Amendment 1092 to the Defense Authorization Act of 2012… [...]

  5. [...] Bunnie had a fantastic overview of some of the issues with Amendment 1092 to the Defense Authorization Act of 2012… Amendment 1092 to the Defense Authorization Act of 2012 is a well-intentioned but misguided provision outlining measures designed to reduce the prevalence of counterfeit chips in the US military supply chain. [...]

  6. HeBD says:

    [...]I also include in this category parts that are new only the sense they have never been soldered onto a board, but were stored improperly (for example, in a humid environment) and should be scrapped, but were subsequently reconditioned and sold like new.[...]

    wow, I never heard of a chip going bad from a humid environment! Have you ever considered bloging about little things like that? Just a short note not a lengthy write-up like above. I think a lot of hobbyist/open source developers would love to read about that kind of thing.

    • bunnie says:

      IC cases, particularly plastic ones, are porous. They absorb water. During normal operation this isn’t a problem because the chip itself is passivated.

      The problem comes during soldering. When the chip is heated, the trapped water turns into steam which expands and can cause some spectacular failures. Packages will sometimes explode due to the trapped water becoming vaporized.

      However, the problem is not always that pronounced. Sometimes the pressure only cracks the die, or if you are dealing with an encapsulated module it can cause internally mounted components to shift. This can lead to a long-term reliability problem.

      As a result, most chips are shipped in vacuum-sealed foil packs with desiccant to prevent water from entering the package before soldering. If a chip is left out in ambient conditions for more than a few hours, it has to be slowly baked to drive off the moisture before being used in production, and even then it’s not a foolproof solution.

      • ewertz says:

        “the trapped water turns into steam which expands and can cause some spectacular failures”

        @HeBD There’s a name for this process, it’s called… wait for it…. “popcorning”.

    • movax says:

      Yep; if you’ve ever ordered BGAs from a vendor like Digi-Key, there’s a sticker on the outside of the bag (which has been vacuum-sealed) indicating a date by which you should re-bake the chips. There’s the “do-not-eat” silica gel desiccant inside, and usually a little card that servers as a humidity indicator as well.

      The bags are usually opaque silver to boot, not the usual ESD bags you see. We always write “please-bake” on parts we send to the assembly house, even if they’re straight from Digi-Key just to be safe (and not waste a $5000 Stratix V).

  7. Rob says:

    Whatever you’re writing about it’s always something fascinating. Thanks!

    (minor point: I think you might have swapped the numbers for $ vs yr penalties, in one paragraph).

  8. [...] On Counterfeit Chips in US Military Hardware [...]

  9. [...] On Counterfeit Chips in US Military Hardware « bunnie’s blog [...]

  10. [...] On Counterfeit Chips in US Military Hardware @ bunnie’s blog. Amendment 1092 to the Defense Authorization Act of 2012 is a well-intentioned but misguided provision outlining measures designed to reduce the prevalance of counterfeit chips in the US military supply chain. [...]

  11. This may be one of the best blog posts I’ve seen written on the subject, especially in the sense of explaining the counterfeit process, and the extent of manufacturing that can go into it. Thank you!

    There is a lot happening now, around the subject. For years, the military was loathe to admit that they were vulnerable to counterfeits. Unfortunately, the growing: availability of e-waste; manufacturing overseas; ease of ghost-shift runs; and OEM’s practices around EOL and obsolescence; economy & budget cuts has all come together in a very “interesting” way.

    I work with a company called GDCA, who specializes in the manufacturing, repair, and support of EOL/obsolete/legacy embedded boards & systems. We’ve worked a lot with military embedded equipment for 20 years, and I’m hoping that the result of this Amendment takes a look at the bigger picture of e-waste & losses in US manufacturing — and not just a “trusted/untrusted” list & supply chain fines.

  12. [...] easy to mark is dubious, as this fascinating letter from Andrew ‘bunnie’ Huang about counterfeit chips in troops hardware explains. Here’s a [...]

  13. [...] are relatively easy to spot is dubious, as this fascinating essay from Andrew 'bunnie' Huang about counterfeit chips in military hardware explains. Here's the background: Amendment 1092 to the Defense Authorization Act of 2012 is a [...]

  14. astroboi says:

    One must wonder if some of the scare documentaries about third world recycling are designed to destroy the component scavenging trade. I once saw a video which supposedly showed how deadly and dangerous it was to recycle electronics. What they did was throw a smoke flare, the sort of thing used to attract rescuers to a downed pilot, into a trash skip. Horrendous thick red smoke poured out and we were told this was an attempt to extract gold! And we were told that poor natives were required to desolder chips with their bare hands with the result that they became cripples within months! Yet a closer look showed clean and well dressed ladies, often with seemingly chubby and healthy toddlers, sorting hardware and men (with all their fingers) easily removing chips from circuit boards. I had to wonder why it was necessary to include so much nonsense in the video and now I think I know. Our government would much prefer all electronic waste to be utterly destroyed or permanently buried. Certainly computer makers would like this too. Who wants a ten year old server running and working when the correct course of action is to buy a brand new one? So once again its “won’t somebody think of the environment” and more rubbish about “carbon footprints” and “think green” when the real objective is to put the scavengers out of business and sell more new electronics.

    • Mike Lindner says:

      I agree with this comment. Smacks of protectionism to me.
      Simple way to avoid the $5Mil penalty, if you’re a military contractor make sure you are damn sure of your supply chain. The way governments over spend on equipment it should be affordable to do this.
      No doubt the strategic reserves are there, I’d doubt there is much risk of getting a dud chip in part for a critical piece of defence hardware, more likely in bulk items like radios and night vision scopes.
      Customs are trained to spot fakes every day, from DVDs to handbags…
      Your final comment on recycling is they key imho, we should all be disgusted that we ship our waste anywhere other than where we live.

  15. [...] On Counterfeit Chips in US Military Hardware [...]

  16. [...] On Counterfeit Chips in US Military Hardware « bunnie’s blog – This article talks about the differnt types of counterfeit parts used in military electronics. I’ts quite interesting. Although military applications are the concern here, I’m also concerned about fake products in the Enterprise One can purchase crudely desoldered MSM7000-series chips (the brains of many Android smartphones) by the pound, at around ten cents for a chip. These chips are then cleaned up, reballed and sometimes remarked, put into tapes and reels and sold as brand new, commanding over a 10x markup. Thus, a single batch of chips can net thousands of dollars, making it a compelling source of income for skilled labor that would otherwise work in a factory for $200 per month. [...]

  17. The thing that arises China as a suspect is when the US military decided to point their fingure, they wanted to send a state senate to China for further investigation.

    This senate visit was declined . . . . I wonder why.

    What is clear is that suppliers and distributers need to be more vigilant on knowig the source of their obsolete components.

  18. cursos de cocina en barcelona…

    [...]On Counterfeit Chips in US Military Hardware « bunnie's blog[...]…