Archive for the ‘chumby’ Category

Interview with MAKE: The End of chumby, New Adventures

Tuesday, May 1st, 2012

Last week, the Internet discovered the end of chumby as you have known it. My exit from the company five months ago was deliberately discreet. It was a good run, but it was also time for me to move on. Upon hearing the news, my good friend Phil Torrone reached out to do an interview, and I was happy to oblige. The interview encapsulates some of my experiences that may be applicable to others excited to get into the hardware business. Here’s some of the questions that I answer for Phil:

  • Can you talk about making a device from start to finish, from idea to factory to retail shelves?
  • What were the challenges with retail sales?
  • Did you get any patents? How do they work within the world of open-source?
  • Do you have any advice for a maker who is considering taking VC funding? Anything different if they’re doing open-source hardware?
  • What are your thoughts on Kickstarter for funding?
  • When you advise companies what do you most often suggest to the founders?
  • If you could do it over, how would you change the hardware of the Chumby? The software? The way Chumby was made?
  • Now that you’ve been part of a full cycle of a VC funded company that makes hardware, what suggestions do you have for company structure, from the people to the location, to the overall organization?
  • What’s next for bunnie, what are most excited about to do next?

    If you’re interested, have a read at the jump!

  • MicroSD card FAQ

    Friday, March 23rd, 2012

    A while back I wrote an analysis of fake microSD cards. As a result of the post, I’ve received this question regularly via email:

    “I’m trying to buy a thousand microSD cards for my embedded controller project. How do you qualify a microSD card?”

    So, I thought it might be helpful to share my answer here.

    There’s this awkward phase between the weekend project (where you buy your microSD card from Best Buy for $20 and have a no-questions return policy) and being Nokia (where you buy the same cards for $2 in quantities large enough to actually have leverage over vendors). When you source a few thousand cards at a time on the wholesale spot market, you’re basically on your own to control quality.

    As far as process control, some vendors are easier to work with than others. Samsung will bump their part numbers based on die revs or other significant internal changes to the card. Sandisk, on the other hand, uses a very short part number for their cards, so you have no idea if the NAND on the inside is MLC or TLC, etc.; you just know the capacity and the card is simply guaranteed to perform to spec. To wit, Sandisk is very thorough about ensuring they meet the spec. However, it’s the edge cases that usually bite you in production; regardless of the spec, every die/controller combo has some character and your embedded controller may bring out some of that color. And, of course, there’s the fakes — Sandisk is a huge target for fakes, people who want to borrow their good name to sell you a batch of shoddy cards.

    If you’re working with a distributor, get a copy of their authorization letter that certifies the relationship with the brand they are selling. It’s easy to fake the certificate, but it’s a good formality to pursue anyways. If you can, get the upstream brand to confirm the distribution relationship.

    Aside from these supply-chain side things, here’s a check-list of technical tests to run on your cards:

    For each new distributor:

    1. I read out the CID and CSD registers and decode them. This is easy to do on linux with a directly connected microSD card. You cannot do this if the card is plugged into a USB adapter — you need to have the card plugged into a direct SD interface. The CID and CSD should look “right” i.e., the manufacturer ID should make sense (unfortunately the manufacture ID codes are all secret, but I can assure you it’s not supposed to be FF or 00), serial numbers should be some big number, date codes correct, etc.

    2. Do a “full write” test at least once. i.e., create a random block of data that’s the putative size of the card, and dd it into the card. Then, do an md5sum of the contents of the card. This will identify loopback tricks that fake capacity. This is a relatively common trick that is surprisingly hard to detect, because many cards are only used to less than 50% capacity in real life.

    3. Do a reboot test, to understand the behavior of the controller/die combo during ungraceful powerdown. It’s less important on systems that can never have their battery removed.

    Before the test, I do a recursive find piped to md5sum to get a full map of all the files in the card. Then, I use a script that writes a random amount of /dev/urandom data in odd-sized blocks (ranging from a couple hundred bytes to a couple megabytes) to the card and then calls sync, in a constant loop after boot. For each block written, the md5sum is recorded. At boot time, all old blocks are checked for md5sum consistency and then deleted. The system under test is automatically power cycled by cutting the AC power about once very 2-3 minutes plus some random interval (depends on how long it takes your device to boot). I cut on the AC power side to capture the effects of the power decay curve of the wall adapter; the logic goes that a clean power down is less likely to cause problems than a gradual powerdown. I run the test on a cohort of at least 2 systems for 2 days straight. If you want to get fancy, you have the system upload its statistics to a server so you can see exactly when it starts to fail. After a couple of days, I extract the card from the system and redo the recursive find with md5sum to verify that no non-critical files have been corrupted that would be difficult to notice without the comprehensive check. Be sure, of course, to ignore files that naturally vary.

    I still don’t have a straight answer on why some cards perform better under this test and others fail miserably. Ultimately, however, every card I’ve encountered eventually corrupts the filesystem after enough cycles, it’s just a matter of how long. I feel comfortable if I can reliably get to ten thousand ungraceful reboots-while-writing before failure. Note that supposedly eMMC has design features that harden cards against these problems, but I’ve never had the luxury of building such high volume systems that eMMC becomes an affordable option. Besides, I consider giving users the ability to remove the firmware card and reflash it with new code using a common USB adapter an important feature, at least in the systems I design. Mobile phone carriers would think differently.

    Of course, once a vendor is qualified, they can still send you bad lots.

    For each new lot I get, I take a few cards and burn them myself and check they boot the system before handing them over the factory. I also manually inspect the CID/CSD to ensure that the manufacturer’s IDs haven’t rotated and I inspect the laser markings to ensure that the lot number changes (it should — if it doesn’t then they are pulling something wonky on you). I also compare the circuit trace pattern on the back, visible through the reliefs in the solder resist coating. If you have easy access to an X-ray machine (some CMs have them on site) you can go so far as to compare the internal construction in the x-ray to see if the dies have been revved. If all these are the same you’re probably good to go on the new lot, but I do pay attention to the failure rate data in the first couple hours of production just to make sure there isn’t something to worry about.

    There’s probably a bunch of other tests, techniques and good ideas that I should be aware of…look forward to reading the comments!

    China: Crowdsourced Tax Enforcement

    Thursday, March 22nd, 2012

    Riddle me this: how does a government enforce tax collection in a cash-only society? Cash has the wonderful property of being anonymous, and therefore hard to track. As a result, cash businesses often under-report revenues, thereby dodging a portion of tax payments.

    China is primarily a cash-driven economy; few local places will accept payment cards of any kind (event rent payments are made in cash — a big, fat stack of cash, as the largest bill in China has an equivalent value of about US$15). As such, China has a big challenge around collecting taxes.

    A solution to the problem is to go with a tax pre-payment system. At the beginning of every month, every business is required to pay an estimated tax. Proof of tax payment is issued in the form of “fapiao” (发票). They look a bit like the one below:

    This fapiao represents tax paid on 10元 (元 is like the $ symbol, and colloquially pronounced “kuai”), so the restaurant I got this from probably paid about 1-2 kuai for this fapiao. When you settle your bill in a restaurant, in addition to getting the itemized receipt, you are supposed to receive a stack of fapiao of equivalent face value.

    At the end of the month, the restaurant claims a tax refund on any remaining fapiao. As a result, fapiao are basically as good as money to the restaurant; hence, the fapiao are printed on watermarked paper with anti-counterfeiting measures, and employ serial numbers you can validate by sending an SMS to a government hotline. Also, restaurants have a strong incentive to omit a few fapiao from your stack, or completely forgo giving you the fapiao (they love it when foreigners dine, because they don’t know about fapiao — they get big business and they get the tax refund on it!).

    So, how does one enforce the distribution of fapiao to customers? China’s clever solution is to make every fapiao a lottery ticket. If you look at the above photo carefully, you’ll see two metallized patches on the fapiao. You can scratch these off, and underneath might reveal a prize! Of course, the one I have above is a losing ticket — it just says “thank you”, with a serial number; but the prize can be thousands of kuai.

    And so, China has crowdsourced tax enforcement, by potentially rewarding citizens with a cash reward for asking for all of their tax pre-payment receipts, and using them up by scratching off the prize areas. The cost of this massive force multiplier is vanishingly small, as all they are offering is the chance to win; I have only ever seen one winning ticket in the past couple of years, and it was for about 2 kuai. Still, it is a nice cultural touch to the end of a big meal, everyone sitting in a circle, scratching their fapiao to see if they won a prize for playing the part of a Chinese tax enforcement agent.

    Of course, with every new system, new problems come in. One is that the waitstaff might nick a couple of fapiao en route to the customer. So now, to get your fapiao you usually have to go in person to a special counter that manages its distribution. And, of course, the restaurant can offer a bribe in place of the fapiao. Just this past month when I was visiting Harbin, I went to collect my lottery tickets and the lady at the register glanced at my 80 kuai receipt and offered to pay me 4 kuai instead of giving me fapiao! I was a bit surprised at how brazen the offer was, but in retrospect, I clearly was not from around there, and thus unlikely to be an auditor.

    On Counterfeit Chips in US Military Hardware

    Saturday, December 3rd, 2011

    Amendment 1092 to the Defense Authorization Act of 2012 is a well-intentioned but misguided provision outlining measures designed to reduce the prevalance of counterfeit chips in the US military supply chain.

    The Defense Authorization Act already has drawn flack for a provision that gives the US military authorization to detain US citizens indefinitely without trial, and I think it rather ironically requires an assessment of the US Federal Debt owed China as a potential “National Security Risk” (section 1225 of HR1540) — anyone want to take bets as to whether the conclusion of this assessment leads to prioritizing deficit reduction as a national security issue, or if it leads to justifying further borrowing from China to build up a military to fend off its biggest creditor?

    Under the proposed anti-counterfeit amendment, first-time offenders can receive a $5 million fine and 20 years prison for individuals, or $15 million for corporations; a penalty comparable to that of trafficking cocaine. While the amendment explicitly defines “counterfeit” to include refurbished parts represented as new, the wording is regrettably vague on whether you must be willfully trafficking such goods to also be liable for such a stiff penalty.

    If you took a dirty but legitimately minted coin and washed it so that it looked mint condition and then sold it to a collector as mint quality, nobody would accuse you of counterfeiting. Yet, this amendment puts a 20 year, $5 million penalty on not only the act of counterfeiting chips destined for military use, but potentially the unwitting distribution of such chips that you putatively bought as new but couldn’t tell yourself if they were refurbished. Unfortunately, in many cases an electronic part can be used for years with no sign of external wear.

    The amendment also has a provision to create an “inspection program”:

    (b) Inspection of Imported Electronic Parts –

    (1) … the Secretary of Homeland Security shall establish a program of enhanced inspection by U.S. Customs and Border patrol of electronic parts imported from any country that has been determined by the Secretary of Defense to have been a significant source of counterfeit electronic parts …

    It’s one thing to inspect fruits and vegetables as they enter the country for pests and other problems; but it is misguided to require Customs officers to become experts in detecting fakes, and/or to burden vendors with the onus of determining whether parts are authentic, particularly with such high penalties involved and the relative ease that forgers can create high-quality counterfeit parts.

    To better understand the magnitude of the counterfeiting problem, it’s helpful to know fakes are made. The fakes I’ve seen fall into the following broad categories:

    1) Trivial external mimicry. Typically these are empty plastic packages with authentic-looking topmarks, or remarked parts that share only physical traits with the authentic parts (for example, a TTL logic chip in an SO-20 case remarked as an expensive microcontroller that uses the same SO-20 case). I consider this technique trivial because it is so easy to detect during factory test; in the worst case you are sold a thin mixture of authentic and conterfeit parts so that testing just one part out of a tube or reel isn’t good enough. However, in all cases the problem is discovered before the product ships so long as the product overall is thoroughly tested.

    2) Refurbished parts. These are authentic parts recovered from e-waste that have been desoldered and reprocessed to appear as new. These are very difficult to spot since the chip is in fact authentic, and a skilled refurbisher can create stunningly authentic-looking results that can only be discriminated with the use of electronic micsoscopes and elemental/isotopic analysis. I also include in this category parts that are new only the sense they have never been soldered onto a board, but were stored improperly (for example, in a humid environment) and should be scrapped, but were subsequently reconditioned and sold like new.

    3) Rebinned parts. These are parts that were authentic, and perhaps have never been used (so can be classified as “new”), but have their markings changed to reflect a higher specification of an identical function. A classic example is grinding and remarking CPUs with a higher speed grade, or more trivially parts that contain lead marked as RoHS-compliant. However, it can get as sophisticated as vendors reverse engineering and reprogramming the fuse codes inside the chip so that the chip’s electronic records match the faked markings on top; or vendors have been known to do deep hacks on Flash drive firmware so that a small memory can appear to a host OS as a much larger memory, going so far as to “loop” memory so that writes beyond the capacity of the device appear to succeed.

    4) Ghost-shift parts. These are parts that are created on the exact same fabrication facility as authentic parts, but run by employees without authorization of the manufacturer and never logged on the books. Often times they are assigned a lot code identical to a legitimate run, but certain testing steps are skipped. These fakes can be extremely hard to detect. It’s like an employee in a mint striking extra coins after-hours.

    5) Factory scrap. Factory rejects and pilot runs can be recovered from the scrap heap for a small bribe, and given authentic markings and resold as new. In order to avoid detection, workers often replace the salvaged scrap with physically identical dummy packages, thus foiling attempts to audit the scrap trail.

    6) Second-sourcing gone bad. Second-sourcing is a standard industry practice where competitors create pin-compatible replacements for popular products in order to create price competition and strengthen the supply chain against events like natural disasters. The practice goes bad when inferior parts are re-marked with the logos of premium brands. High-value but functionally simple discrete analog chips such as power regulators are particularly vulnerable to this problem. Premium US brands can command a 10x markup over Asian brands, as “drop-in replacement” Asian-brand parts are notorious for spotty quality, cut corners and poor parametric performance. However, there is a lot of money to be made buying blanks from the second source fab and remarking them with authentic-looking top marks of premium US brands. In some cases there are no inexpensive or fast tests to detect these fakes, short of decapsulating the chip and comparing mask patterns and cross-sections.

    In the case of the US Military, they have a unique problem where they are one of the biggest and wealthiest buyers of really old parts. Military designs have shelf lives of decades, but parts have production cycles of only years. It’s like asking someone to build a NeXT Cube motherboard today using only certifiably new parts; no secondhand or refurbished parts allowed. I don’t think it’s possible.

    The impossibility of this situation may force military contractors to be complicit in the consumption of counterfeit parts. For example, the fake parts in the P-8 Poseidon were “badly refurbished”. A poor refurbishing job is probably detectable with a simple visual inspection, so even though people are quick to point fingers at China, maybe part of the problem is that the contractor was lax in checking incoming stock — or perhaps looking the other way, because if these are the last parts of its kind in the world, what else can they do?

    Another part of the senate hearings revealed that L3 bought counterfeit video memory chips destined for C-27J aircraft from Global IC Trading Group. Well … duh. Global IC ain’t Digikey … they specialize in trading excess, overruns and secondhand goods. The prices are often good, but I only go to them if I’m really in a bind, and I’m willing to accept odd lots to get production moving at any cost. L3′s big enough to have a professional sourcing group aware of that, and thus exercise extreme caution when buying from such vendors.

    My guess is that the stocks of any distributor in the secondhand electronics business are already flooded with undetected counterfeits. Remember, only the bad fakes are ever caught, and chip packaging was not designed with anti-counterfeiting measures in mind. While all gray market parts are suspect, that’s not necessarily a bad thing. Gray markets play an essential role in the electronics ecosystem; using them is a calculated but sometimes unavoidable risk.

    While the situation is clearly a mess now, some simple measures going forward could help fix things for the future. One could involve embedding anti-counterfeit measures in chips approved for military use. For chips larger than 1cm, a unique 2-D barcode can be applied with laser markings. The equipment to do such laser-marking is relatively commonplace today in chip packaging facilities. The efficacy of such techniques has been proven in biotech, where systems such as Matrix 2D are deployed to track disposable sample tubes in biology labs. Despite a tiny footprint, the codes are backed with a guarantee of 100% uniqueness. Another potential solution is to mix a UV dye into the component’s epoxy that changes fluorescence properties upon exposure to reflow temperatures. If the dye is distributed through the plastic body of the case, the change will be impossible to remove with grinding alone.

    A second partial measure could be to manage e-waste better. E-waste is harvested in bulk for used parts. One can purchase crudely desoldered MSM7000-series chips (the brains of many Android smartphones) by the pound, at around ten cents for a chip. These chips are then cleaned up, reballed and sometimes remarked, put into tapes and reels and sold as brand new, commanding over a 10x markup. Thus, a single batch of chips can net thousands of dollars, making it a compelling source of income for skilled labor that would otherwise work in a factory for $200 per month.

    If we stopped shipping our e-waste overseas for disposal, or at least ground up the parts before shipping them over, then the feedstock for such markets would be reduced. It could also create jobs domestically for processing the e-waste, which by the way is a source of gold comparable to the richest gold ore. On the other hand, I’m of the opinion that in the big picture this sort of component-level recycling is actually quite good for the environment and the human ecosystem. Upon disposal, most electronics still have years of serviceable life in them, and there is a hungry market for technology in emerging economies that cannot be met with new parts purchased on the primary market.

    A final option could be to establish a strategic reserve of parts. A production run of military planes is limited to perhaps hundreds of units, and so I imagine the lifetime demand of a part including replacements is limited to tens of thousands of units. I can fit ten thousand chips in the volume of a large shoebox; at least physically, it’s not an unmanageable proposition. These are small volumes compared to consumer electronics volumes. I imagine that purchasing a reserve of raw replacement components for critical avionics systems would only add a fraction of a percent to the cost of an airplane, and could even lead to long term cost savings as manufacturers can achieve greater scale efficiency if they run one large batch all at once. This could be a foolproof method to ensure supply trustability for critical military hardware.

    Random Stories from China

    Tuesday, November 15th, 2011

    I’ve just returned from my first vacation in China. I came to the realization earlier this year that despite routine visits to China, I had only seen one tiny part of China — mostly the insides of factories in Guandong province. I’ve heard that China is much bigger than that, and so Chuang Tzu’s little frog decided it was time to poke its head above the well and see how big the sky is. My perlfriend and I picked a couple spots far, far away from factories. I wanted to go to Harbin, but we couldn’t find convenient flights, so instead we went to Yangshuo and Chongqing. Chances are you’ve never heard of them — and that’s exactly why we decided to visit them.

    Yangshuo and its peasants tending rice paddies was an interesting study on China’s past; Chongqing was an interesting study on China’s future. According to wikipedia, the municipality of Chongqing has a population of 28 million people (that’s more than the entirety of Australia), yet few of my western friends have heard of it. It’s a special administrative zone, directly managed by Beijing; non-ironic pictures and busts of chairman Mao were common fixtures. This is in contrast to Shenzhen’s western influences, in part due to its proximity to Hong Kong. Chongqing is a hotbed of investment and growth; I’ve heard the city mentioned numerous times in rumors of mass factory migrations from Guandong to central China, where costs are lower. The city of Chongqing is definitely a lot cheaper than Shenzhen, even for a tourist; everything from cabs to food to hotel was cheaper by a substantial margin. While the air quality was terrible in Chongqing, overall I had to say I was pleasantly surprised to find that the citizens of Chongqing had a remarkably … wholesome … feel about them. They felt more laid back than Shenzheners. Kids played in the street. Teens would congregate at night in the plaza near my hotel and hang out, but were well behaved even into the wee hours. One could hardly walk a block without seeing people playing Mahjong in the street on makeshift card tables. My interactions with shop vendors were friendly, and often pleasant; bargaining was easier, and the service was generally patient and helpful. I suppose part of this might be due to the fact that a foreigner is still a novelty in Chongqing; in two days I saw only two western-looking people, and few spoke English. It probably also helps that Chongqing’s population is more local, with fewer seasonal workers and estranged immigrants.

    I normally don’t make fun of Engrish — partially because there’s so much Engrish in China it’s hard to know where to start, and partially because my Budong Hua (get it? 普通话 (putong hua) = “Mandarin” vs. 不懂话 (budong hua) = “not understanding language”…the Mandarin equivalent of “Engrish”) is so bad I’m throwing stones from a glass house. However, this one item from a menu at a hotpot place called “qi huo guo” (chongqing is supposedly the origin of hotpot) really caught my inner nerd’s eye.

    Given a choice between pig brains, ox throat or “odd bosons” (奇包子), I’d be most concerned about ingesting food made from strange fundamental particles; at least the others aren’t potentially made of antimatter. Thankfully, the odd bosons aren’t a product of some LHC experiment gone awry. “Odd bosons” are simply the restaurant’s eponymous dumplings (qi huo guo means “odd hot pot”). On recommendation by the waitstaff, I ordered the odd bosons. They were delicious.

    On my way back to Singapore, I stopped through Shenzhen and gave Star Simpson an introductory tour of the hua qian bei markets. While wandering I finally found a book I had long been searching for:

    Yep, that’s right, the book of iPhone schematics. I snapped that baby up for $4. Unfortunately, the pages are bigger than my flatbed scanner, so I could only capture about 80% of a page. Here’s a snippet from the table of contents:

    Inside are a few handy diagrams, such as a component layout guide:

    And also a list of key footprints with interesting pins highlighted:

    And of course, pages and pages of schematics:

    My feeling is that these schematics probably come from leaks of original Apple sources, because many of the annotations couldn’t be divined from a clean-room reverse engineering job. For example, the above schematics annotate that the AP_UART connection on the dock has a dual-footprint option for a possible drop-in DisplayPort upgrade. Anyways, these schematics are useful as a sourcing guide for cheap components. Any part found in this book has been made in millions-per-week quantities, which is a handy fact to keep in mind when bargain hunting for stable supplies of cheap components.

    A classy addition to this book is a full-color teardown guide, with photos courtesy of ifixit:

    While the book doesn’t credit ifixit for their labors, there are few things more validating of ifixit’s world-class status than Shanzhai copying ifixit’s materials into the canon of phone repair guides. Mad props to my peeps at ifixit.

    Another little gem I bought in the market are the metal stencils pictured below.

    These stencils, purchased at less than $1 each, are for various mobile phone chipsets. The stencils come in this “summary” form, and also in a more useful knock-out library where the metal is pre-cut around the edges of the footprints so you can clip out a single footprint and use it to guide the application of solderpaste to a PCB. However, my suspicion is that these “summary” form stencils are probably not used for applying solderpaste to mainboards, but rather used to identify chips that have been pulled off of boards through mass desoldering, and also for use in reballing the same BGAs. I’ve seen similar stencils used with great efficacy to manually reball BGAs on a factory line in Guanzhou. Hand-reballing of BGAs is surprisingly fast and efficient with the aid of a stencil and a machined jig. I remember watching with prurient fascination as the operator reballed BGA after BGA in a matter of minutes; mental arithmetic placed the cost of reballing at around a dime. Thus refurbished, the recycled chips can be used to repair broken phones, or to build whole “new” phones from scratch.