Archive for the ‘Ponderings’ Category

Safecast Geiger Counter Reference Design

Thursday, March 15th, 2012

This past weekend marked the anniversary of the Tohoku-Oki earthquake that devastated Japan. I had not felt my blood so cold since I watched the twin towers fall almost a decade earlier. I still vividly remember the twisting knots I felt in my stomach as I watched the footage of a tsunami wiping out huge swathes of Japanese countryside. In a matter of hours, entire cities were washed off the map, leaving an eerie post-apocalyptic landscape of a few survivors weeping amongst twisted wreckage. Then, in the ensuing days, Fukushima Daiichi melted down, leaving in its wake one of the worst on-going radiation contamination crisis since Chernobyl.

I have good friends in Japan, and I visit often. I wanted to do something to help, but I didn’t know what I could do. I was connected by Joi Ito to Safecast, and I joined the effort to build an open sensor network that could aggregate trustable, source-neutral radiation monitoring data. Safecast itself has many talented and hard working volunteers who have done a remarkable job of achieving their goals by instrumenting Japan with radiation monitors and aggregating data through cleverly designed and rapidly deployable mobile monitoring capabilities.

I decided my tiny contribution to the effort would be to design a radiation monitor suitable for everyday civilian use. This is a preventative/preparedness measure, addressing the long-term issue of empowering a civilian population with few available options for power generation to self-monitor their environment. The problem with the current crop of radiation monitors is that they are basically laboratory instruments: accurate & reliable, but bulky, expensive, and difficult to use, requiring a degree in nuclear physics to understand exactly what the readings meant. Another problem with crises like these is that while radiation monitoring is important, it’s something that is typically neglected by the civilian population until it is too late.

Therefore, the challenge set out before me was to design a new Geiger counter that was not only more intuitive and easier to use than the current crop, but was also sufficiently stylish so that civilians would feel natural carrying it around on a daily basis. Furthermore, it had to provide extensive logging capabilities, as radiation monitors are typically not turned “on” until after the fact. It also had to operate effectively in catastrophic conditions, i.e. in scenarios where internet and power have been cut for days. Finally, the data collected by the instrument had to pass any scrutiny thrown its way, and the collected data had to be traceable to a given instrument so that if its calibration is incorrect, its data can be selectively excluded without poisoning the entire dataset. Radiation monitoring is a politically sensitive subject, and certain parties have interests to manipulate the data one way or the other to promote their views with the public. Ad-hoc data collection networks suffer from the possibility that their efforts can be discredited by institutions with big budgets who find that the readings represent an inconvenient truth.

Radiation sensing primer

Radiation measurements are subtle, partly because radiation comes in many flavors. Many Geiger counters can only efficiently detect the most energetic kind of radiation, gamma radiation. This includes the Geiger counters frequently favored by government and regulatory agencies. However, there are weaker forms of radiation (alpha and beta) which often go overlooked that can also pose a human health risk, particularly if they are ingested or inhaled. These weaker forms of radiation are also by-products of a nuclear meltdown, and because they come from different isotopes they have different patterns of distribution and absorption in the environment.

Because of the diversity of radiation sources and their varying biological impact, it is very hard to determine if an environment is safe in the face of an elevated Geiger counter reading. However, improved historical and spatial distribution records of background radiation measurements can help identify when there is a spike in radiation, which is a clearer cause for concern.

In the interest of creating a complete solution for public health needs, a core design requirement of the new Geiger counter is to incorporate a sensor that could detect all three forms of radiation. This type of sensor is a “pancake” style Geiger tube, which has a large mica window that enables sensitivity to all three kinds of radiation. The ultimate selection of the LND7317 pancake tube plus iRover HV radiation sensing core influenced every aspect of the industrial design (ID) and internal electronics.

There and Back Again: a Hacker’s Tale

I thought it would be interesting to share not only the final design, but also the intermediate designs that were scrapped en route to achieving a final design. Design is an iterative process, where one has to make difficult choices about what to include and more significantly what to leave out. It’s extremely rare to see what got left on the cutting room floor, but I saved my notes along the way so I could share them with you now.

Initial Design Sketch

Above is a rendering of the first design sketch, made back when Safecast had the name of “RDTN”. I do all my industrial design using Solidworks, a survival skill I picked up during my tenure at chumby designing consumer electronics. I came up with this in the first couple of weeks after the disaster. This design incorporated a low-sensitivity tube from Sparkfun, because at that time I did not understand the importance of using a pancake tube.

The biggest problem I wanted to solve with this design is user abandonment. Radiation leaks are thankfully rare events. However, this also means that when an event happens, there is typically a lack of pre-crisis background data against which to compare the post-crisis readings. Therefore, I wanted to build a device that people would be compelled to carry around every day and use for years at a time.

My thought is that the average consumer would have a hard time justifying carrying around yet another gadget in their pockets or purses for the sole purpose of measuring radiation. Within weeks or even days of getting nothing but “safe” readings, the Geiger counter would be forgotten and left at home to languish until after the next crisis.

One way to compel users to carry around a Geiger counter is to put it into something they already carry all the time. While it would be tough to convince a smartphone vendor to incorporate a very expensive and bulky Geiger tube, many smartphone users also carry around a spare battery pack, which they use almost every day. So, I thought it might be a good idea to trojan horse a Geiger tube into such a battery pack.

The sketch above demonstrates such an incarnation. This design is basically a battery pack that can charge a smartphone, but also incorporates a Geiger tube, an LED flashlight (handy in an emergency when there is no power), and some logging circuitry. The Geiger counter would upload its log data to the Safecast network whenever a user plugged in to charge the phone.

The design itself is minimalist, with a shape inspired by the steam cooling towers frequently used to iconify nuclear power in western media. The shape was chosen to remind us that sometimes we have no choice but to harvest the power of the atom, and a well-equipped and informed civilian data collection network is a key factor in trusting the safety of our power sources.

A second iteration

The first sketch had to be abandoned, primarily because the sensor it was designed around was too small to effectively measure alpha and beta radiation. After Safecast settled on the LND7317 Geiger tube as the standard reference sensor, I started re-designing the sensor around the new tube.

The problem with the larger, more sensitive sensor is that it was big – over a half-inch thick, and a couple inches in diameter. Below is a sketch of a design study aimed at creating the smallest possible Geiger counter that could also incorporate the large pancake tube. It’s about the size of a hockey puck, but a bit thicker. In order to keep the size and weight of the Geiger counter reasonable, I had to abandon any notion of a dual-purpose as a battery pack. Instead, I had to rely on “sex appeal” alone to compel users to carry the device around. I wanted to make the Geiger counter something unique and aesthetically pleasing, something you would enjoy carrying around frequently. I started from a minimalist design – the puck – and endeavored to design-out any outward indicators or displays. Hence in this sketch, the radiation measurement is provided by a set of super-high efficiency 7-segment LEDs that could shine the numbers through a seemingly opaque white shell. The design’s shape and feel was meant to be somewhere in between “Eve” from WALL-E and an egg.

Unfortunately, this design, too, had to be abandoned because at the time when I was drawing up the sketches, I didn’t have detailed mechanical drawings of the LND7317 tube. When I was finally given a sample of the tube and drawings for it, I discovered there was not only the puck-like body, but also a nearly 1″ long protrusion for the cathode and anode. This completely destroyed any notion of building a puck-like sensor.

Closing in on the final design

Below is a rendering of an attempt to accommodate the accurate CAD model for the LND7317 into an ID that stayed faithful to the Eve/egg design inspirations. The puck was elongated to the minimum dimensions required to house all the internal components. Again, the hidden 7-segment LED display motif was employed.

The final design

After much discussion and review with the Safecast team, we decided that a key component of the user experience should be a graphic display, instead of a 7-segment LED readout. Therefore, a 128×128 pixel OLED panel was incorporated into the design. The OLED panel would be mounted behind a continuous outer shell, so there would be no seams or outward design features resulting from the introduction of the OLED. However, as the OLED is not bright enough to shine through an opaque white plastic exterior shell, a clear window had to be provided for the OLED. As a result, the naturally black color of the OLED caused the preferred color scheme of the exterior case to go from light colors to dark colors. User interaction would occur through a captouch button array hidden behind the same shell, with perhaps silkscreen outlines to provide hints as to where the buttons were underneath the shell. I had originally resisted the idea of using the OLED because it’s very expensive, but once I saw how much an LND7317 tube would cost in volume, I realized that it would be silly to not add a premium feature like an OLED. Due to the sensor alone, the retail price of the device would be in the hundreds of dollars; so adding an OLED display would help make the device “feel” a lot more valuable than a 7-segment LED display, even though the OLED’s presence is largely irrelevant to the core function of the apparatus.

The design also lacks any integrated radio connection. A popular request for the design was the incorporation of a bluetooth or zigbee style radio; however, a combination of a very stringent battery life goal (several months of standby time) and low manufacturing volumes meant that it was impractical to incorporate a radio into the device. It’s a slippery slope to start adding features like GPS and bluetooth – to add those features, you’d need to upgrade the microcontroller, at which point you’re basically building a very expensive, heavy and large cell phone with a geiger counter in it. Furthermore, the entire development effort was being done by an unpaid volunteer operating on a shoestring budget – Safecast isn’t Apple. So, rather than build a buggy cell phone that can sense radiation, I’d rather build an outstanding Geiger counter; hence the decision to focus efforts and resources on core functionality, with the sole allowance being the inclusion of the OLED + captouch array for improved UI. This is a controversial design decision and I fully expect to be chastised for it.

The Prototypes

Once the design was finished, the next step was to build prototypes. This is the really fun part, where you turn your ideas into something you can touch and hold.

The prototypes are made out of CNC-machined ABS (even the clear part!). The cosmetic moldings that go over the connectors were also built and they do fit, but because of their expense and fragility (CNC milled ABS lacks the robustness of injection-molded ABS), I try not to install them, even for glamor shots. To wit, the whole thing was done on a shoestring budget, as Safecast is a non-profit; two full prototypes were built, including PCB fab, assembly, and CNC milling for one and a half revisions of the cases, for a bit under $3k total.

An important point readers should note about this design is that I’m not manufacturing this Geiger counter reference design. My contribution is limited to design IP only. Practically speaking, I’d make a terrible Geiger counter supplier, because I don’t have the credibility or history in the industry. Instead, the design has been donated to the community, thereby enabling International Medcom, a business that has spent decades specialized in producing high-quality Geiger counters, to bring this to the market. If you’re interested in getting one of these, keep an eye on their website.

The final design features include:

  • LND7317 pancake tube + iRover HV board
  • STM32-based microcontroller; sufficient CPU power to digitally sign logs with a unique private key as a non-repudiation/anti-tamper measure
  • 450 mAh Li-poly battery
  • 3-axis accelerometer so sensor orientation can be recorded
  • 128×128 color OLED display
  • 6-button captouch array
  • “hold” button on the back to lock the captouch array and prevent false triggering of the power-hungry UI elements
  • lanyard attachment (important for the Japanese market)
  • microUSB port for charging and data upload interface, featuring an FTDI-based serial chipset capable of loading firmware into the microcontroller
  • 3.5mm jack capable of bidirectional audio
  • embedded hall-effect sensor (to detect attachments, e.g. for occluding alpha or beta radiation)
  • audible event notification via piezo buzzer
  • low-power visual event notification via conventional LED
  • real-time clock
  • a high-quality entropy source ;-)
  • I am a proponent of open source hardware; so here’s the source files for my design! All of the following source files are licensed under CC3.0-BY-SA with my XL1.0 automatic patent cross-license rider (CC doesn’t address patents, so I invented my own rider that piggybacks on CC to ensure that any patents that may arise from this or its derivatives are automatically cross-licensed to the community).

  • Altium design source / schematics / gerbers / BOM for the mainboard electronics
  • Altium design source / schematics / gerbers / BOM for the buttonboard electronics
  • Solidworks design source / IGES / STEP for the industrial design
  • For those who don’t have 3D design tools, you can install Solidwork’s free e-drawings viewer and look at the easm file, or if you run windows you can download this executable and just run it
  • Of course, a hardware prototype is only the beginning – there’s a huge amount of effort remaining on the software. To bootstrap things, xobs and I have coded up a core demonstration system based on Leaf Lab’s libmaple. You can peruse the code and/or download it at github. Basically, this demo system provides an architecture to easily register drivers and facilitate power management. The validation demo shown running on the prototype photos above indicate that all of the hardware features work. But, the software has yet to have a layer of polish and shine added in terms of the UI and power management optimization.

    A key design goal electronics’ system design was to enable community participation. As such, I eschewed the use of JTAG adapters during development. Instead, hooks were provided in the hardware to enable the integrated FTDI USB-serial controller to flash the microcontroller’s firmware via a “bitbang” interface. As a result, anyone who has an interest in developing for this Geiger counter can simply plug it into their laptop’s USB port and start coding without any need for proprietary JTAG adapters or proprietary software to purchase, as the entire developer’s toolchain is available in source form. We were able to code up and test the entire functionality demo (including sleep/stop/standby power management) using nothing more than the USB-serial capability built into the design. As I write this, I realize I had neglected to upload the firmware loader to github, so here’s a tarball for it; currently, the loader only runs under Linux and OSX.

    I think there’s some fun things the community could do with the UI on a Geiger counter. At the very least, the microcontroller has sufficient power to play Tetris. Another whimsical thought was to build a subsystem that would play music out the audio port based upon the current radiation level — calm, ambient music in low-radiation environments escalating to death metal and the sound track of “Run Lola Run” at dangerous levels.

    So that’s it! I hope that the design ultimately helps the people of Japan – or people anywhere in the world where radiation contamination may be a concern – to feel more empowered and in control of their situation.

    “The Amp Hour” Podcast

    Tuesday, February 28th, 2012

    I was a guest on this week’s The Amp Hour radio show. It was good fun having a chat with show hosts Dave Jones and Chris Gammell — talked about everything from PCB design at 35,000 feet to life in Asia to NeTV and the legal challenges of system design.

    Thanks for the Support

    Saturday, February 11th, 2012

    Thanks to everyone who signed my petition to the copyright office in support of the EFF’s DMCA exemption requests. I’m overwhelmed that over 25,000 signatures were collected in just a fortnight.

    As a supplement to the public petition, I submitted a second letter with personal comments to the copyright office as well. It’s all public record, and here’s a copy of it in case you’re interested in what I had to say:

    Dear Ms. Pallante,

    I am writing in support of proposed exemption classes 3 (consoles), 4 (personal computing devices) and 5 (smartphones and tablets). I am separately filing a petition with about 25,000 signatures of like-minded consumers who also support these exemption requests.

    I’ve been on many sides of the DMCA equation – as a developer, a researcher and a user. In the past, I conducted research on the Microsoft Xbox, which led ultimately to a jailbreak for the platform. This happened while I was earning my PhD in electrical engineering at MIT. In addition, as a hobbyist I have tinkered with many platforms, including the Sega Dreamcast, the Nintendo Gamecube, and various printers. Today, my principal business is the development and sales of end-user hardware devices, many of which incorporate cryptographic elements to control access and to maintain privacy.

    I have seen a contraction of user’s rights over my two decades in the industry, and a corresponding decline in small business innovation. I would like to ensure the law recognizes and upholds users’ rights with respect to privately owned hardware. These rights enable users to innovate and improve their hardware. I refer to game consoles, tablets, smartphones, and other computing devices in the same breath because as a hardware designer, all of these systems look basically identical on the inside.

    The exemptions I am supporting help preserve a traditional notion of ownership. Before the DMCA, there was a clean, bright line at the checkout counter of a store. Once you bought it, it was yours – if you wanted to use it as a doorstop, you could; if you wanted to modify it to do something more useful, that was also fine. If there was a bug or a security problem with the device, you were free to patch it. Without these exemptions, the owner’s right to modify, repair, improve and develop for devices that incorporate now-ubiquitous cryptographic technology is hampered by the potential legal ramifications of jailbreaking.

    Repairing your broken game console is a specific example of something that’s risky to do on your own because of the DMCA. As units age, parts wear out and need replacing. For example, the Xbox has a hard drive; a hard drive will wear out after a few years, rendering the console inoperable. Thus, a user who has invested hundreds of dollars into a game library will eventually suffer its loss of use. Due to the Xbox’s security measures, replacing the hard drive requires jailbreaking the console. Without an exemption, users will ultimately be left with the choice of either abandoning their investment, or facing potential legal problems in the course of repairing or seeking the repair of their console.

    Of course, for a limited period of time, one option is to buy a new or second-hand console; or if the platform supports it, purchase a next-gen console which incorporates emulation of previous-generation games. However, the popularity of retro-gaming today demonstrates that certain game titles have a timeless appeal, and can be playable for generations. Nintendo’s Mario franchise is deeply embedded in game culture partly because 80′s vintage consoles can be repaired and played again by both new and old gamers. However, the DMCA makes it risky to engage in basic console repairs that require jailbreaks, such as replacing a worn-out hard drive. In essence, retro-gaming ends where the DMCA begins.

    The right to jailbreak is also critical for innovation. As a hardware developer and businessman, I know that the developers don’t always get it right. Locking down a platform turns the dialog between users and companies into a one-sided monologue; the DMCA creates legal uncertainty for users who want to look under the hood and develop an informed opinion about their technology. While the dialog between companies and users is not always harmonious, the result in the end is generally a better solution for the general public. Within 3 days of the A5 iPhone jailbreak, almost 1 million users downloaded the jailbreak. There are over 2 million users of the Nintendo Wii who have installed the home brew channel, enabling them to play independently developed video games. And there are at least hundreds of thousands of Xbox users who run the XBMC application, and millions who have downloaded the same app for their PCs.

    XBMC originally stood for the “Xbox Media Center”. It was developed on jailbroken Xboxes in response to the limitations of the Microsoft-sanctioned media player application. Microsoft was slow in responding to user requests for an enhanced media playback experience; furthermore, Microsoft media products tended to favor their proprietary formats, limiting user choice. However, due to cryptographic access controls implemented in the hardware, users could not simply write their own media player application. Therefore, a jailbreak had to be developed – exposing the developers to potential legal liability. However, the passion of these developers is reflected in the quality of the software. Today, the XBMC application has migrated from consoles to PCs, runs on everything from Macs to PCs to set top boxes, and has even resulted in new startups, such as Boxee, based around the platform technology. Without a jailbreak, none of this would have come to pass.

    There is an incredible reservoir of innovation that is being held back by the DMCA dam, and granting an exemption to consoles, smartphones, tablets, and other personal computing devices will unleash a powerful torrent of new ideas and improvements.

    Please grant these exemptions to re-enable grass-roots innovation and to preserve the traditional notion of ownership.

    Sincerely,

    Andrew “bunnie” Huang, PhD

    You Bought It, but Do You Own It?

    Thursday, January 26th, 2012

    On February 10th, I’m sending a letter to the Library of Congress in support of granting exemptions to the DMCA for jailbreaking your own devices. If you believe that you should be able to run whatever programs you want on your own hardware, please sign my letter to show support; anyone from anywhere in the world can sign. You can also submit your own letter to the Library of Congress, if you feel so inclined or disagree with my opinions.

    In 2002, I intercepted a key on the original Xbox that allowed me to encrypt and run my own software on the device. Even though that Xbox had a Pentium processor on the inside — the same CPU found in my desktop PC — without that key, I could only run the limited selection of software provided to me by Microsoft.

    When I was informed about the DMCA, which became law in 1998, it was a bucket of cold water thrown at my face; I felt deeply disenfranchised. You see, I was a graduate student at MIT at the time, and up until that point the freedom to create, explore, and overcome barriers was encouraged, even celebrated. It was bewildering that running linux on this PC with the green X is illegal, yet running linux on this architecturally identical beige box next to it was legal. A chill descended upon the situation; MIT sent letters to me officially repudiating involvement in my activities, fearing the worst. Fortunately, brave souls at the MIT AI lab stood up for me in defiance of the campus counsel, and provided me with resources and the connections to the EFF to negotiate with Microsoft and see a positive ending to the whole situation.

    I’m lucky. Not everyone has the encouragement, wisdom and strength of a team of MIT faculty and EFF counsel behind them. Without further exemptions to the DMCA enabling jailbreaking, freedom to innovate and tinker withers. Since then, many lawsuits have been filed under the DMCA, creating a tone of fear. Research projects are abandoned, business plans are scrapped; and the stalwart operators left with the will to research jailbreaks work in shadow, a constant fear of lawsuit haunting them for the mere practice of attempting to load their own software onto hardware that they legally own. Entrepreneurs and innovators should not be so burdened, especially at a time when we need their valuable contributions to bootstrap new businesses.

    I believe if you buy hardware, you should own it; and ownership means nothing less of full rights to do with it as you wish. If you believe in this too, please sign my letter to the Library of Congress in support of extended exemptions to the DMCA, enabling jailbreaks for more platforms.

    A special thanks to the EFF for preparing the website and helping me with the letter!

    On Counterfeit Chips in US Military Hardware

    Saturday, December 3rd, 2011

    Amendment 1092 to the Defense Authorization Act of 2012 is a well-intentioned but misguided provision outlining measures designed to reduce the prevalance of counterfeit chips in the US military supply chain.

    The Defense Authorization Act already has drawn flack for a provision that gives the US military authorization to detain US citizens indefinitely without trial, and I think it rather ironically requires an assessment of the US Federal Debt owed China as a potential “National Security Risk” (section 1225 of HR1540) — anyone want to take bets as to whether the conclusion of this assessment leads to prioritizing deficit reduction as a national security issue, or if it leads to justifying further borrowing from China to build up a military to fend off its biggest creditor?

    Under the proposed anti-counterfeit amendment, first-time offenders can receive a $5 million fine and 20 years prison for individuals, or $15 million for corporations; a penalty comparable to that of trafficking cocaine. While the amendment explicitly defines “counterfeit” to include refurbished parts represented as new, the wording is regrettably vague on whether you must be willfully trafficking such goods to also be liable for such a stiff penalty.

    If you took a dirty but legitimately minted coin and washed it so that it looked mint condition and then sold it to a collector as mint quality, nobody would accuse you of counterfeiting. Yet, this amendment puts a 20 year, $5 million penalty on not only the act of counterfeiting chips destined for military use, but potentially the unwitting distribution of such chips that you putatively bought as new but couldn’t tell yourself if they were refurbished. Unfortunately, in many cases an electronic part can be used for years with no sign of external wear.

    The amendment also has a provision to create an “inspection program”:

    (b) Inspection of Imported Electronic Parts –

    (1) … the Secretary of Homeland Security shall establish a program of enhanced inspection by U.S. Customs and Border patrol of electronic parts imported from any country that has been determined by the Secretary of Defense to have been a significant source of counterfeit electronic parts …

    It’s one thing to inspect fruits and vegetables as they enter the country for pests and other problems; but it is misguided to require Customs officers to become experts in detecting fakes, and/or to burden vendors with the onus of determining whether parts are authentic, particularly with such high penalties involved and the relative ease that forgers can create high-quality counterfeit parts.

    To better understand the magnitude of the counterfeiting problem, it’s helpful to know fakes are made. The fakes I’ve seen fall into the following broad categories:

    1) Trivial external mimicry. Typically these are empty plastic packages with authentic-looking topmarks, or remarked parts that share only physical traits with the authentic parts (for example, a TTL logic chip in an SO-20 case remarked as an expensive microcontroller that uses the same SO-20 case). I consider this technique trivial because it is so easy to detect during factory test; in the worst case you are sold a thin mixture of authentic and conterfeit parts so that testing just one part out of a tube or reel isn’t good enough. However, in all cases the problem is discovered before the product ships so long as the product overall is thoroughly tested.

    2) Refurbished parts. These are authentic parts recovered from e-waste that have been desoldered and reprocessed to appear as new. These are very difficult to spot since the chip is in fact authentic, and a skilled refurbisher can create stunningly authentic-looking results that can only be discriminated with the use of electronic micsoscopes and elemental/isotopic analysis. I also include in this category parts that are new only the sense they have never been soldered onto a board, but were stored improperly (for example, in a humid environment) and should be scrapped, but were subsequently reconditioned and sold like new.

    3) Rebinned parts. These are parts that were authentic, and perhaps have never been used (so can be classified as “new”), but have their markings changed to reflect a higher specification of an identical function. A classic example is grinding and remarking CPUs with a higher speed grade, or more trivially parts that contain lead marked as RoHS-compliant. However, it can get as sophisticated as vendors reverse engineering and reprogramming the fuse codes inside the chip so that the chip’s electronic records match the faked markings on top; or vendors have been known to do deep hacks on Flash drive firmware so that a small memory can appear to a host OS as a much larger memory, going so far as to “loop” memory so that writes beyond the capacity of the device appear to succeed.

    4) Ghost-shift parts. These are parts that are created on the exact same fabrication facility as authentic parts, but run by employees without authorization of the manufacturer and never logged on the books. Often times they are assigned a lot code identical to a legitimate run, but certain testing steps are skipped. These fakes can be extremely hard to detect. It’s like an employee in a mint striking extra coins after-hours.

    5) Factory scrap. Factory rejects and pilot runs can be recovered from the scrap heap for a small bribe, and given authentic markings and resold as new. In order to avoid detection, workers often replace the salvaged scrap with physically identical dummy packages, thus foiling attempts to audit the scrap trail.

    6) Second-sourcing gone bad. Second-sourcing is a standard industry practice where competitors create pin-compatible replacements for popular products in order to create price competition and strengthen the supply chain against events like natural disasters. The practice goes bad when inferior parts are re-marked with the logos of premium brands. High-value but functionally simple discrete analog chips such as power regulators are particularly vulnerable to this problem. Premium US brands can command a 10x markup over Asian brands, as “drop-in replacement” Asian-brand parts are notorious for spotty quality, cut corners and poor parametric performance. However, there is a lot of money to be made buying blanks from the second source fab and remarking them with authentic-looking top marks of premium US brands. In some cases there are no inexpensive or fast tests to detect these fakes, short of decapsulating the chip and comparing mask patterns and cross-sections.

    In the case of the US Military, they have a unique problem where they are one of the biggest and wealthiest buyers of really old parts. Military designs have shelf lives of decades, but parts have production cycles of only years. It’s like asking someone to build a NeXT Cube motherboard today using only certifiably new parts; no secondhand or refurbished parts allowed. I don’t think it’s possible.

    The impossibility of this situation may force military contractors to be complicit in the consumption of counterfeit parts. For example, the fake parts in the P-8 Poseidon were “badly refurbished”. A poor refurbishing job is probably detectable with a simple visual inspection, so even though people are quick to point fingers at China, maybe part of the problem is that the contractor was lax in checking incoming stock — or perhaps looking the other way, because if these are the last parts of its kind in the world, what else can they do?

    Another part of the senate hearings revealed that L3 bought counterfeit video memory chips destined for C-27J aircraft from Global IC Trading Group. Well … duh. Global IC ain’t Digikey … they specialize in trading excess, overruns and secondhand goods. The prices are often good, but I only go to them if I’m really in a bind, and I’m willing to accept odd lots to get production moving at any cost. L3′s big enough to have a professional sourcing group aware of that, and thus exercise extreme caution when buying from such vendors.

    My guess is that the stocks of any distributor in the secondhand electronics business are already flooded with undetected counterfeits. Remember, only the bad fakes are ever caught, and chip packaging was not designed with anti-counterfeiting measures in mind. While all gray market parts are suspect, that’s not necessarily a bad thing. Gray markets play an essential role in the electronics ecosystem; using them is a calculated but sometimes unavoidable risk.

    While the situation is clearly a mess now, some simple measures going forward could help fix things for the future. One could involve embedding anti-counterfeit measures in chips approved for military use. For chips larger than 1cm, a unique 2-D barcode can be applied with laser markings. The equipment to do such laser-marking is relatively commonplace today in chip packaging facilities. The efficacy of such techniques has been proven in biotech, where systems such as Matrix 2D are deployed to track disposable sample tubes in biology labs. Despite a tiny footprint, the codes are backed with a guarantee of 100% uniqueness. Another potential solution is to mix a UV dye into the component’s epoxy that changes fluorescence properties upon exposure to reflow temperatures. If the dye is distributed through the plastic body of the case, the change will be impossible to remove with grinding alone.

    A second partial measure could be to manage e-waste better. E-waste is harvested in bulk for used parts. One can purchase crudely desoldered MSM7000-series chips (the brains of many Android smartphones) by the pound, at around ten cents for a chip. These chips are then cleaned up, reballed and sometimes remarked, put into tapes and reels and sold as brand new, commanding over a 10x markup. Thus, a single batch of chips can net thousands of dollars, making it a compelling source of income for skilled labor that would otherwise work in a factory for $200 per month.

    If we stopped shipping our e-waste overseas for disposal, or at least ground up the parts before shipping them over, then the feedstock for such markets would be reduced. It could also create jobs domestically for processing the e-waste, which by the way is a source of gold comparable to the richest gold ore. On the other hand, I’m of the opinion that in the big picture this sort of component-level recycling is actually quite good for the environment and the human ecosystem. Upon disposal, most electronics still have years of serviceable life in them, and there is a hungry market for technology in emerging economies that cannot be met with new parts purchased on the primary market.

    A final option could be to establish a strategic reserve of parts. A production run of military planes is limited to perhaps hundreds of units, and so I imagine the lifetime demand of a part including replacements is limited to tens of thousands of units. I can fit ten thousand chips in the volume of a large shoebox; at least physically, it’s not an unmanageable proposition. These are small volumes compared to consumer electronics volumes. I imagine that purchasing a reserve of raw replacement components for critical avionics systems would only add a fraction of a percent to the cost of an airplane, and could even lead to long term cost savings as manufacturers can achieve greater scale efficiency if they run one large batch all at once. This could be a foolproof method to ensure supply trustability for critical military hardware.