Archive for the ‘Ponderings’ Category

I Broke My Phone’s Screen, and It Was Awesome

Sunday, May 25th, 2014

So this past week has been quite a whirlwind — we wrapped up the Novena campaign and smashed all our stretch goals, concluding with over $700k raised, and I got my hair cut in a bar at midnight by none other than the skilled hands of Lenore of Evil Mad Scientist Laboratories (I blame Jake! :). It was an exhilarating week; xobs and I are really grateful for the outpouring of support and we’re looking forward to working with the community to build an open hardware ecosystem that can grow for years to come.

On my way back home to Singapore, I stopped by Dongguan to have a visit with my supply chain partners to hammer out production plans for Novena. Unfortunately, as I was getting out of the taxi at the Futian border checkpoint going into China, I dropped my phone on the sidewalk and shattered its screen.

There is no better place in the world to break your phone’s screen than the border crossing into Shenzhen. Within an hour of dropping the phone, I had a new screen installed by skilled hands in Hua Qiang Bei, for a price of $25.

Originally, I thought I would replace the screen myself — on my broken phone, I hastily visited iFixit for details on the procedure to replace the screen, and then booked it over to Hua Qiang Bei to purchase the replacement parts and tools I would need. The stall I visited quoted me about US$120 for a new screen, but then the lady grabbed my phone out of my hands, and launched a built in self test program on the phone by dialing *#0*# into the phone dialer UI.

She confirmed that there were no bad pixels on my OLED display and that the digitizer was still functional, but just cracked. She then offered to buy my broken OLED+digitizer assembly off of me, but only if they did the work to replace my screen. I said it would be fine as long as I could watch them do the job, to make sure they aren’t swapping out any other parts on me.

They had no problem with that, of course — so my phone came apart, had the old broken OLED+digitizer assembly separated, adhesive stripped from the phone body, replaced with a proper new film of adhesive, a “new” (presumably refurbished) OLED+digitizer fitted and re-assembled in 20 minutes. The whole service including parts and labor came out to $25. I kept on thinking “man I should take pictures of this” but unfortunately the device I would use to take said pictures was in pieces in front of me. But, I’ll hint that the process involved a hair dryer (used as a heat gun), copious amounts of contact cleaner (used to soften the adhesive on the OLED+digitizer module), and a very long thumbnail (in lieu of a spudger/guitar pick).

This is the power of recycling and repair — instead of paying $120 for a screen and throwing away what is largely a functional piece of electronics, I just had to pay for the cost of just replacing the broken glass itself. I had originally assumed that the glass on the digitizer is inseparable from the OLED, but apparently those clever folks in Hua Qiang Bei have figured out an efficient method to recycle these parts. After all, the bulk of the assembly’s cost is in the OLED display, and the touchscreen sensor electronics (which are also grafted onto the module) are also undamaged by the fall. Why waste perfectly good parts, anyways?

And so, my phone had a broken screen for all of an hour, and it was fixed for less than the cost of shipping spare parts to Singapore. There is no better place to break your phone than in Shenzhen!

Circuit Stickers Manufacturing Retrospective: From Campaign to First Shipment

Tuesday, April 29th, 2014

Last December, Jie Qi and I launched a crowdfunding campaign to bring circuit stickers under the brand name of “chibitronics” to the world.

Our original timeline stated we would have orders shipped to Crowd Supply for fulfillment by May 2014. We’re really pleased that we were able to meet our goal, right on time, with the first shipment of over a thousand starter kits leaving the factory last week. 62 cartons of goods have cleared export in Hong Kong airport, and a second round of boxes are due to leave our factory around May 5, meaning we’ve got a really good chance of delivering product to backers by Mid-May.

Above: 62 cartons containing over a thousand chibitronics starter kits waiting for pickup.

Why On-Time Delivery Is So Important
A personal challenge of mine was to take our delivery commitment to backers very seriously. I’ve seen too many under-performing crowdfunding campaigns; I’m deeply concerned that crowdfunding for hardware is becoming synonymous with scams and spams. Kickstarter and Indiegogo have been plagued by non-delivery and scams, and their blithe caveat emptor attitude around campaigns is a reflection of an entrenched conflict of interest between consumers and crowdfunding websites: “hey, thanks for the nickel, but what happened to your dollar is your problem”.

I’m honestly worried that crowdfunding will get such a bad reputation that it won’t be a viable platform for well-intentioned entrepreneurs and innovators in a few years.

I made the contentious choice to go with Crowd Supply in part because they show more savvy around vetting hardware products, and their service offering to campaigns — such as fulfillment, tier-one customer support, post-campaign pre-order support, and rolling delivery dates based on demand vs. capacity — is a boon for hardware upstarts. Getting fulfillment, customer support and an ongoing e-commerce site as part of the package essentially saves me one headcount, and when your company consists of just two or three people that’s a big deal.

Crowd Supply doesn’t have the same media footprint or brand power that Kickstarter has, which means it is harder to do a big raise with them, but at the end of the day I feel it’s very important to establish an example of sustainable crowdfunding practices that is better for both the entrepreneur and the consumer. It’s not just about a money grab today: it’s about building a brand and reputation that can be trusted for years to come.

Bottom line is, if I can’t prove to current and future backers that I can deliver on-time, I stand to lose a valuable platform for launching my future products.

On-Time Delivery Was not Easy
We did not deliver chibitronics on time because we had it easy. When drawing up the original campaign timeline, I had a min/max bounds on delivery time spanning from just after Chinese New Year (February) to around April. I added one month beyond the max just to be safe. We ended up using every last bit of padding in the schedule.

I made a lot of mistakes along the way, and through a combination of hard work, luck, planning, and strong factory relationships, we were able to battle through many hardships. Here’s a few examples of lessons learned.

A simple request for one is not necessarily a simple request for another. Included with every starter kit is a fantastic book (free to download) written by Jie Qi which serves as a step-by-step, self-instruction guide to designing with circuit stickers. The book is unusual because you’re meant to paste electronic circuits into it. We had to customize several aspects of the printing, from the paper thickness (to get the right light diffusion) to the binding (for a better circuit crafting experience) to the little pocket in the back (to hold swatches of Z-tape and Linqstat material). Most of these requests were relatively easy to accommodate, but one in particular threw the printer for a loop. We needed the metal spiral binding of the book to be non-conductive, so if someone accidentally laid copper tape on the binding it wouldn’t cause a short circuit.

Below is an example of how a circuit looks in the book — in this case, the DIY pressure sensor tutorial (click on image for a larger version).

Checking for conductivity of a wire seems like a simple enough request for someone who designs circuits for a living, but for a book printer, it’s extremely weird. No part of traditional book printing or binding requires such knowledge. Because of this, the original response from the printer is “we can’t guarantee anything about the conductivity of the binding wire”, and sure enough, the first sample was non-conductive, but the second was conductive and they could not explain why. This is where face to face meetings are invaluable. Instead of yelling at them over email, we arranged a meeting with the vendor during one of my monthly trips to Shenzhen. We had a productive discussion about their concerns, and at the conclusion of the meeting we ordered them a $5 multimeter in exchange for a guarantee of a non-conductive book spine. In the end, the vendor was simply unwilling to guarantee something for which he had no quality control procedure — an extremely reasonable position — and we just had to educate the vendor on how to use a multimeter.

To wit, this unusual non-conductivity requirement did extend our lead time by several days and added a few cents to the cost of the book, but overall, I’m willing to accept that compromise.

Never skip a checkplot. I alluded to this poignant lesson with the following tweet:


The pad shapes for chibitronics are complex polyline geometries, which aren’t handled so gracefully by Altium. One problem I’ve discovered the hard way is the soldermask layer occasionally disappears for pads with complex geometry. One version of the file will have a soldermask opening, and in the next save checkpoint, it’s gone. This sort of bug is rare, but it does happen. Normally I do a gerber re-import check with a third-party tool, but since this was a re-order of an existing design that worked before, and I was in a rush, I skipped the check. Result? thousands of dollars of PCBs scrapped, four weeks gone from the schedule. Ouch.

Good thing I padded my delivery dates, and good thing I keep a bottle of fine scotch on hand to help bitter reminders of what happens when I get complacent go down a little bit easier.

If something can fit in a right and a wrong way, the wrong way will happen. I’m paranoid about this problem — I’ve been burned by it many times before. The effects sticker sheet is a prime example of this problem waiting to happen. It is an array of four otherwise identical stickers, except for the LED flashing pattern they output. The LED flashing pattern is controlled by software, and trying to manage four separate firmware files and get them all loaded into the right spot in a tester is a nightmare waiting to happen. So, I designed the stickers to all use exactly the same firmware; their behaviors set by the value of a single external resistor.

So the logic goes: if all the stickers have the same firmware, it’s impossible to have a “wrong way” to program the stickers. Right?

Unfortunately, I also designed the master PCB panels so they were perfectly symmetric. You can load the panels into the assembly robot rotated by pi radians and the assembly program runs flawlessly — except that the resistors which set the firmware behavior are populated in reverse order from the silkscreen labels. Despite having fiducial holes and text on the PCBs in both Chinese and English that are uniquely orienting, this problem actually happened. The first samples of the effects stickers were “blinking” where it said “heartbeat”, “fading” where it said “twinkle”, and vice-versa.

Fortunately, the factory very consistently loaded the boards in backwards, which is the best case for a problem like this. I rushed a firmware patch (which is in itself a risky thing to do) that reversed the interpretation of the resistor values, and had a new set of samples fedexed to me in Singapore for sanity checking. We also built a secondary test jig to add a manual double-check for correct flashing behavior on the line in China. Although, in making that additional test, we were confronted with another common problem –

Some things just don’t translate well into Chinese. When coming up with instructions to describe the difference between “fading” (a slow blinking pattern) and “twinkling” (a flickering pattern), it turns out that the Chinese translation for “blink” and “twinkle” are similar. Twinkle translates to 闪烁 (“flickering, twinkling”) or 闪耀 (to glint, to glitter, to sparkle), whereas blink translates to 闪闪 (“flickering, sparkling, glittering”) or 闪亮 (“brilliant, shiny, to glisten, to twinkle”). I always dread making up subjective descriptions for test operators in Chinese, which is part of the reason we try to automate as many tests as possible. As one of my Chinese friends once quipped, Mandarin is a wonderful language for poetry and arts, but difficult for precise technical communications.

Above is an example of the effects stickers in action. How does one come up with a bulletproof, cross-cultural explanation of the difference between fading (on the left) and twinkling (on the right), using only simple terms anyone can understand, e.g. avoiding technical terms such as random, frequency, hertz, periodic, etc.

After viewing the video, our factory recommended to use “渐变” (gradual change) for fade and “闪烁” (flickering, twinkling) for twinkle. I’m not yet convinced this is a bulletproof description, but it’s superior to any translation I could come up with.

Funny enough, it was also a challenge for Jie and I to agree upon what a “twinkle” effect should look like. We had several long conversations on the topic, followed up by demo videos to clarify the desired effect. The implementation was basically tweaking code until it “looked about right” — Jie described our first iteration of the effect as “closer to a lightning storm than twinkling”. Given the difficulty we had describing the effect to each other, it’s no surprise I’m running into challenges accurately describing the effect in Chinese.

Eliminate single points of failure. When we built test jigs, we built two copies of each, even though throughput requirements demanded just one. Why? Just in case one failed. And guess what, one of them failed, for reasons as of yet unknown. Thank goodness we built two copies, or I’d be in China right now trying to diagnose why our sole test jig isn’t working.

Sometimes last minute changes are worth it. About six weeks ago, Jie suggested that we should include a stencil with the sensor/microcontroller kits. She reasoned that it can be difficult to lay out the copper tape patterns for complex stickers, such as the microcontroller (featuring seven pads), without a drawing of the contact patterns. I originally resisted the idea — we were just weeks away from finalizing the order, and I didn’t want to delay shipment on account of something we didn’t originally promise. As Jie is discovering, I can be very temperamental, especially when it comes to things that can cause schedule slips (sorry Jie, thanks for bearing with me!). However, her arguments were sound and so I instructed our factory to search for a stencil vendor. Two weeks passed and we couldn’t find anyone willing to take the job, but our factory’s sourcing department wasn’t going to give up so easily. Eventually, they found one vendor who had enough material in stock to tool up a die cutter and turn a couple thousand stencils within two weeks — just barely in time to meet the schedule.

When I got samples of the sensor/micro kit with the stencils, I gave them a whirl, and Jie was absolutely right about the utility of the stencils. The user experience is vastly improved when you have a template to work from, particularly for the microcontroller sticker with seven closely spaced pads. And so, even though it wasn’t promised as part of the original campaign, all backers who ordered the sensor/micro kit are getting a free stencil to help with laying out their designs.

Chinese New Year has a big impact the supply chain. Even though Chinese New Year (CNY) is a 2-week holiday, our initial schedule essentially wrote off the month of February. Reality matched this expectation, but I thought it’d be helpful to share an anecdote on exactly how CNY ended up impacting this project. We had a draft manuscript of our book in January, but I couldn’t get a complete sample until March. It’s not because the printer was off work for a month straight — their holiday, like everyone else’s, was about two weeks long. However, the paper vendor started its holiday about 10 days before the printer, and the binding vendor ended its holiday about 10 days after the printer. So even though each vendor took two weeks off, the net supply chain for printing a custom book was out for holiday for around 24 days — effectively the entire month of February. The staggered observance of CNY is necessary because of the sheer magnitude of human migration that accompanies the holiday.

Shipping is expensive, and difficult. When I ran the initial numbers on shipping, one thing I realized is we weren’t selling circuit stickers — at least by volume and weight, our principle product is printed paper (the book). So, to optimize logistics cost, I was pushing to ship starter kits (which contain a book) and additional stand-alone book orders by ocean, rather than air.

We actually had starter kits and books ready to go almost four weeks ago, but we just couldn’t get a reasonable quotation for the cost of shipping them by ocean. We spent almost three weeks haggling and quoting with ocean freight companies, and in the end, their price was basically the same as going by air, but would take three weeks longer and incurred more risk. It turns out that freight cost is a minor component of going by ocean, and you get killed by a multitude of surcharges, from paying the longshoreman to paying all the intermediate warehouses and brokers that handle your goods at the dock. All these fixed costs add up, such that even though we were shipping over 60 cartons of goods, air shipping was still a cost-effective option. To wit, a Maersk 40′ sea container will fit over 1250 cartons each containing 40 starter kits, so we’re still an order of magnitude away from being able to efficiently utilize ocean freight.

We’re not out of the Woods Yet. However excited I am about this milestone, I have to remind myself not to count my chickens before they hatch. Problems ranging from a routine screw-up by UPS to a tragic aviation accident to a logistics problem at Crowd Supply’s fulfillment depot to a customs problem could stymie an on-time delivery.

But, at the very least, at this point we can say we’ve done everything reasonably within our power to deliver on-time.

We are looking forward to hearing our backer’s feedback on chibitronics. If you are curious and want to join in on the fun, the Crowd Supply site is taking orders, and Jie and I will be at Maker Faire Bay Area 2014, in the Expo hall, teaching free workshops on how to learn and play with circuit stickers. We’re looking forward to meeting you!

Qué romántico!

Wednesday, October 30th, 2013

Nothing says “I love you” quite like a fake ON-semi 16-pin SOIC.

Mitch Davis sent me this photo, posted in a Chinese-only trade group chat room for chip sellers in Huaqiangbei. The poster said, “Does anyone know who supplies this chip, my customer needs it urgently!”

I figure if you can put any fake markings on any chip, this would be a romantic way to give a sly wink to that girl in the material quality inspection office you’ve had eyes on. Now all we need are “will you marry me” chips: “Hey darling, can you help me rework this board? I can’t quite make out the part number on this chip…” Now the hard part is, what chip would be most appropriate for the big question?

From Spark: Why Kickstarters are Always Delayed

Tuesday, October 1st, 2013

Zach Supalla, Founder and CEO at Spark, wrote a frank introspective on why Kickstarters are always delayed. His thoughts are particularly germane, as he and his team are currently working hard to deliver on the Spark Core’s Kickstart campaign promise. They have taken an ultra-transparent approach to updating supporters on their progress, and their challenges — an approach that takes a lot of courage and thick skin.

You can read his thoughts here.

Dust, Tecate, and Solder: A Hacker’s Holiday (Burning Man 2013)

Sunday, September 15th, 2013

I went to Burning Man for the second time this year. I had an amazing time hanging out with so many interesting and fun people. I spent many a night dancing to the beats of the Robot Heart again, taking in 85,000 watts of pure, uncut bass: it’s where silicon touches the soul. If you haven’t been to Burning Man, this article gives the most balanced perspective I’ve seen to date.

This year I came early for the so-called “Working Man”: the days before Burning Man’s official start, when the art is built and the camps are pitched. As always, looking behind the curtain takes away a bit of the magic, but a lot of things made more sense to me. I chuckle a bit at my naiveté last year, when I thought the artwork appearing on the playa through the week was some sort of organized performance art experience. In reality, it’s just people running behind schedule and working feverishly through the night to get huge monuments erected just in time to be burned into ashes.

This year, as in my previous year, I camped with a group called The Phage, which is part of a three-camp complex known as The Institute (the other two camps are Relaxomatic and False Profit). The camp doesn’t happen without a lot of planning and preparation, but coming from Singapore, it’s hard to be very useful in pre-camp prep. So I did what I could do — make circuit boards.

The Phage camp’s logo is a stylized version of the bacteriophage virus, so I made a custom-styled “locator badge” for our camp this year.

I also made, more or less on a whim, a small driver board for WS2812B-style LED strips.

Below is a short movie of the two in action (sorry IE & Safari users, I only do Ogg Theora).

It’s a unique experience to see your creations being picked up, loved, and then utterly abused for an entire week. It’s even more amazing to get feedback from such friendly, understanding and intelligent users. I spent several splendid afternoons with a soldering iron, teaching people how to solder, as well as fixing and prodding various bits of hardware. As an old-school hacker there are few things more relaxing than sitting around on a lazy afternoon, drinking beers, soldering, and making new friends — and not having to worry about rushing off to my next meeting, completely free of the distractions of the Internet and mobile phones.

Phage Locator Badges

In addition to being a source of light at night, the badges have a “locator” feature. Every badge contains a 433 MHz radio transceiver. On top of this, I built a broadcast paging system so badge-wearers can press a button and cause other people’s badges to strobe in response.

I also attached a high-gain receiver to our art car, Strangelove, so some of the art car’s LED strips would also strobe when the car’s receiver detected a locate request from a badge.

There’s also a huge flame-shooting art piece on our frontage called the “Hive Queen” (on the left in the photo above). Normally, users have to walk up to it and hit buttons to make it shoot fire. Well, after a few beers, someone suggested it would be a good idea to hack another receiver into the Hive Queen (thank you Sean Stevens for loaning me the relay breakout board!). It required a few code changes to ensure that nobody could jam the fire control solenoids on, but it worked.

It’s the first time I’ve ever built a circuit that intentionally caused fire; there’s something viscerally satisfying about remote-controlling six flame throwers at once.

Only 50 of such badges will ever be made, but I figured I’d share the design files so others thinking about doing something similar could benefit. Click the various links for schematics (PDF), gerbers, and PCB source (Altium). You can also find all the firmware for the radio and blinky lights in the Phage git repo.

Compact LED Strip Drivers

The intent behind the LED strip driver was to provide a compact and minimal way to drive cut-to-length LED strips, so that no programming is required at the point of installation. An important feature is the ability to automatically measure the length of an LED strip loop, and adjust the pre-programmed light patterns accordingly. The board was made out of flex circuit material and included a microphone circuit that you could cut off with a pair of scissors, giving an extra dimension of customization with minimal tools.

I was happy to see the number of creative ways people used the LED strip drivers — they were worn as belts, around hats, zip tied to bikes, bags, etc. It was also very instructive to observe how these things failed — at Burning Man, the strips were thoroughly used; they had to survive a week of trudging around the desert, and the repetitive stress of hours of dancing.

I’m also sharing the source for the LED strip driver I made: schematics (PDF), gerbers, and PCB source (Altium). Firmware can be found at this github repo. As a final note, I wouldn’t recommend anyone actually fabricate the designs as-is — please consider them reference material only. There are some flaws in both designs, particularly the flex LED driver. Many things went right, but I would also do many things differently now that I’ve seen how people use them. But, for those who have never done a flex PCB before, the files do give you a taste of some of the quirks that go into making a flexible PCB.

Photo credits: Thanks to Princess SRB, and to Justin Jach for sharing the Phage camp photoset on flickr!