See you at Maker Faire Bay Area!

May 17th, 2014

Looking forward to seeing everyone at Maker Faire Bay Area, happening May 17 & 18 at the San Mateo Event Center. xobs and I will be giving a short half-hour talk starting at 10:30AM in the Expo hall on Saturday about Novena, on the Electronics stage. Afterwards, xobs will be hanging out with his Novena at the Freescale booth, also in the Expo hall, about halfway down on the left hand side across from the Atmel/Arduino booth. If you’re curious to see it or just want to stop by and say hi, we welcome you!

Also, the whole chibitronics crew will be in the Expo hall as well, in the second row between Sony, PCH, and Qualcomm (‽‽‽). We’ll be teaching people how to craft circuits onto paper; attendees who can score a first-come, first-serve spot will receive free circuit stickers and also get a chance to be instructed by the wonderful and dynamic creative genius behind chibitronics, Jie Qi.

Novena in the X-Ray

May 13th, 2014

Last week, Nadya Peek from MIT’s CBA gave me the opportunity to play with their CT scanner. I had my Novena laptop with me, so we extracted the motherboard and slapped it into the scanner. Here are some snapshots of the ethernet jacks, which are enclosed metal boxes and thus a target for “intervention” (e.g. NSA ANT FIREWALK featuring their nifty TRINITY MCM).

Plus, it’s just fun to look at X-rays of your gear.

The X-ray reveals the expected array of ferrite cores implementing the transformers required by gigabit ethernet.

An Oscilloscope Module for Novena

May 8th, 2014

One of Novena’s most distinctive features is its FPGA co-processor. An FPGA, or Field Programmable Gate Array, is a sea of logic gates and memory elements that can be wired up according to hardware descriptions programmed in languages such as Verilog or VHDL. Verilog can be thought of as a very strictly typed C where every line of the code executes simultaneously. Thus, every bit of logic in Novena’s Spartan 6 LX45 FPGA could theoretically perform a computation every clock cycle — all 43,000 logic cells, 54,000 flip flops, and 58 fixed-point multiply accumulate DSP blocks. This potential for massive parallelism underlies one half of the exciting prospects enabled by an FPGA.

The other exciting half of an FPGA relates to its expansive I/O capabilities. Every signal pin of an FPGA can be configured to comply with a huge range of physical layer specifications, from vanilla CMOS to high-speed differential standards such as TMDS (used in HDMI) and SSTL (used to talk to DDR memories). Each signal pin is also backed by a high speed SERDES (serializer/deserializer) and sophisticated clock management technologies. Need a dozen high-precision PWM channels for robotics? No problem, an FPGA can easily do that. Need an HDMI interface or two? Also no problem. Need a bespoke 1000 MT/s ADC interface? Simple matter of programming – and all with the same set of signal pins.

Novena also hangs a 2Gbit DDR3 memory chip directly off the FPGA. The FPGA contains a dedicated memory controller that talks DDR3 at a rate of 800MT/s over a 16-bit bus, yielding a theoretical peak memory bandwidth of 12.8 Gbits/s. This fast, deep memory is useful for caching and buffering data locally.

Thus, the FPGA can be thought of as the ultimate hardware hacking primitive. In order to unlock the full potential of the FPGA, we decided to bring most of the spare I/Os on the chip to a high speed expansion header. The high speed header is a bit less convenient than Arduino shield connectors if all you need to do is flash an LED, but as a trade-off the header is rated for signal speeds of over a gigabit per second per pin.

However, the GPBB (General Purpose Breakout Board) featured as one of the Novena crowdfunding campaign stretch goals resolves this inconvenience by converting the high speed signal format into a much lower performance but more convenient 0.1” pin header format, suitable for most robotics and home automation projects.

Enter the Oscilloscope
A problem that xobs and I frequently encounter is the need for a highly programmable, travel-friendly oscilloscope. There’s a number of USB scope solutions that don’t quite cut it in terms of analog performance and UX, and there are no self-contained solutions we know of today that allow us to craft stimulus-response loops of the type needed for fuzzing, glitching, power analysis, or other similar hardware hacking techniques.

Fortunately, Novena is an ideal platform for implementing a bespoke oscilloscope solution – which we’ve gone ahead and done. Here’s a video demonstrating the basic functionality of our oscilloscope solution running on Novena (720p version in VP8 or H.264):

Novena was plugged into the large-screen TV via HDMI to make filming the video a little bit easier.

In a nutshell, the oscilloscope offers two 8-bit channels at 1GSPS or one 8-bit channel at 2GSPS with an analog bandwidth of up to 900MHz. As a side bonus we also wired in a set of 10 digital channels that can be used as a simple logic analyzer. Here’s some high resolution photos of the oscilloscope expansion board:

Here’s the schematics.

This combination of the oscilloscope expansion board plus Novena is a major step toward the realization of our dream of a programmable, travel-friendly oscilloscope. The design is still a couple revisions away from being production ready, but even in its current state it’s a useful hacking tool.

At this point, I’m going to geek out and talk about the tech behind the implementation of the oscilloscope board.

Oscilloscope Architecture
Below is a block diagram of the oscilloscope’s digital architecture.

The FPGA is configured to talk to an ADC08D1020 dual 1GSPS ADC, designed originally by National Semiconductor but now sold as TI. The interface to the ADC is a pair of 8-bit differential DDR busses, operating at up to 500MHz, which is demultiplexed 1:8 into a 64-bit internal datapath. Upon receipt of a trigger condition, the FPGA stores a real-time sample data from the ADC into local DDR3 memory, and later on the CPU can stream data out of the DDR3 memory via the Linux Generic Netlink API. Because the DDR3 memory’s peak bandwidth is only 1.6GSPS, deep buffer capture of 256 Msamples is only available for net sample rates below 1GSPS; higher sample rates are limited to the internal memory capacity of the FPGA, still a very usable 200 ksamples depth. The design is written in Verilog and consumes about 15% of the FPGA, leaving plenty of space for implementing other goodies like digital filters and other signal processing.

The ADC is clocked by an Analog Devices AD9520 PLL, which derives its time base from a TCXO. This PLL + TCXO combination gives us better jitter performance than the on-chip PLL of the FPGA, and also gives us more flexibility on picking sampling rates.

The power system uses a hybrid of boost, buck, and inverting switching regulators to bring voltages to the minimum-dropout required for point-of-use LDOs to provide clean power to sensitive analog subsystems. This hybrid approach makes the power system much more complex, but helps keep the power budget manageable.

Perhaps the most unique aspect of our oscilloscope design is the partitioning of the analog signal chain. Getting a signal from the point of measurement to the ADC is a major engineering challenge. Remarkably, the same passive probe I held in the 90′s is still a standard workhorse for scopes like my Tektronix TDS5104B almost a quarter century later. This design longevity is extremely rare in the world of electronics. With a bandwidth of several hundred MHz but an impedance measured in mega-ohms and a load capacitance measured in picofarads, it makes one wonder why we even bother with 50-ohm cables when we have stuff like oscilloscope probes. There’s a lot of science behind this, and as a result well-designed passive probes, such as the Tektronix P6139B, cost hundreds of dollars.

Unfortunately, high quality scope probes are made out of unicorn hair and unobtanium as far as I’m concerned, so when thinking about our design, I had to take a clean-sheet look at the problem. I decided to look at an active probe solution, whilst throwing away any notion of backward compatibility with existing scope probes.

I started the system design by first considering the wires (you can tell I’m a student of Tom Knight – one of his signature phrases is “it’s the wires, stupid!”). I concluded the cheapest high-bandwidth commodity cable that is also rated for a high insertion count is probably the SATA cable. It consists of two differential pairs and it has to support signal bandwidths measured in GHz, yet it costs just a couple of bucks. On the downside, any practical probing solution needs to present an impedance of almost a million times greater than that required by SATA, to avoid loading down the circuitry under test. This means we have to cram a high performance amplifier into a PCB that fits in the palm of your hand. Thankfully, Moore’s Law took care of that in the intervening decades from when passive oscilloscope probes were first invented out of necessity.

The LMH6518 is a single-chip solution for oscilloscope front-ends that is almost perfect for this scenario. It’s a 900 MHz, digitally controlled variable gain amplifier (VGA) with the added feature of an auxilliary output that’s well-suited for functioning as a trigger channel; conveniently, a SATA cable has two differential pairs, so we allocate one for measurement and one for trigger. We also strap a conventional 8-pin ribbon cable to the SATA cable for passing power and I2C.

The same LMH6518 VGA can be combined with a variety of front-end amplifiers to create a range of application-specific probes. We use a 1GHz FET op-amp (the ADA4817) to do the impedance transformation required of a “standard” digital oscilloscope. We use a relatively low impedance but “true differential” amplifier to measure voltages developed across a series sense resistor for power signature analysis. And we have a very high-impedance, high CMRR instrumentation amplifier front end for capturing signals developed across small loops and stubs of wire, useful for detecting parasitic electromagnetic emissions from circuits and cables.

Above: digital probe

Above: power signature analysis probe

Above: sidechannel emissions probe

However, the design isn’t quite perfect. The LMH6518 burns a lot of power – a little over a watt; and the pre-amp plus power regulators add about another watt overall to the probe’s power footprint. Two watts isn’t that bad on an absolute scale, but two watts in the palm of your hand is searing hot; the amplifier chip gets to almost 80C. So, I designed a set of custom aluminum heatsinks for the probes to help spread and dissipate the heat.

When I handed the aluminum-cased probes to xobs, I warned him that the heat sinks are either going to solve the heat issue, or it’s going to turn the probes into a ball of flaming hot metal. Unfortunately, the heatsink gets to about 60C in still air, which is an ergonomic challenge – the threshold for pain is typically around 45-50C, so it’s very uncomfortable to hold the aluminum cases directly. It’s alright to hold the probes by the plastic connectors on the back, but this requires special training and users will instinctively want to hold a probe by its body. So, probably I’ll have to do some thermal optimization of the design and add either a heat pipe to a large heatsink off the probe body, or use a small fan to force air over the probes. It turns out just a tiny bit of airflow is all that’s need to keep the probes cool, but with passive convection alone they are simply too hot to handle. This won’t, of course, stop us from using them as-is; we’re okay with having to be a little bit careful to gain access to a very capable device. However, nanny-state laws and potentially litigious customers make it too risky to sell this solution to end consumers right now.

Firmware Architecture

xobs defined the API for the oscilloscope. The driver is based upon the Generic Netlink API native to the Linux kernel, and relies upon the libnl-genl libraries for the user-space implementation. Out of the various APIs available in the Linux kernel to couple kernelspace to userspace, Netlink was the best match, as it is stream-oriented and inherently non-blocking. This API has been optimized for high throughput and low latency, since it is also the core of the IP network stacks that on servers push gigabits of bandwidth. It’s also more mature than the nascent Linux IIO subsystem.

In the case of xobs’ driver, he creates a custom generic netlink protocol which he registers with the name “kosagi-fpga”. Generic netlink sockets support the concept of specific commands, and he currently supports the following:


/* list of valid commands */
enum kosagi_fpga_commands {
KOSAGI_CMD_UNSPEC,
KOSAGI_CMD_SEND,
KOSAGI_CMD_READ,
KOSAGI_CMD_POWER_OFF,
KOSAGI_CMD_POWER_ON,
KOSAGI_CMD_FPGA_ASSERT_RESET,
KOSAGI_CMD_FPGA_DEASSERT_RESET,
KOSAGI_CMD_TRIGGER_SAMPLE,
__KOSAGI_CMD_MAX,
};

The current implementation provisions two memory-mapped address spaces for the CPU to communicate with the FPGA, split along two different chip select lines. Chip Select 0 (CS0) is used for simple messages and register settings, while Chip Select 1 (CS1) is used for streaming data to and from the FPGA. Therefore, when the CPU wants to set capture buffer sizes, trigger conditions, or initiate a transfer, it communicates using CS0. When it wants to stream data from the FPGA, it will do so via CS1.

The core of the API is the KOSAGI_CMD_TRIGGER_SAMPLE and KOSAGI_CMD_READ commands. To request a sample from the oscilloscope, the userspace program emits a KOSAGI_CMD_TRIGGER_SAMPLE command to the kosagi-fpga Netlink interface. This will cause the CPU to communicate with the FPGA via the CS0 EIM memory space control registers, setting up the trigger condition and the transfer FIFO from the FPGA.

The userspace program will then emit a KOSAGI_CMD_READ command to retrieve the data. Upon receiving the read command, the kernel initiates a burst read from CS1 EIM memory space to a kernel buffer using memcpy(), which is forwarded back to the userspace that requested the data using the genlmsg_unicast() Netlink API call. Userspace retrieves the data stream from the kernel by calling the nl_recv() API call.

This call is currently configured to block until the data is available for the userspace program, but it can also be configured to timeout as well. However, a timeout is generally not necessary as the call will succeed in a fraction of a millisecond due to the high speed and determinism of the transfer interface.

In addition to handling data transfers, the kernel module implementing this API also handles housekeeping functions, such as configuring the FPGA and controlling power to the analog front end. FPGA configuration is handled automatically upon driver load (via insmod, modprobe, or udev) via the request_firmware() API built into the Linux kernel. The FPGA bitstream is located in the kernel firmware directory, usually /lib/firmware/novena_fpga.bit.

Power management functions have their own dedicated Netlink commands. Calling these commands causes the respective GPIO for the expansion connector power switch to be toggled. When the expanion connector is power-cycled, the module also resets the FPGA and reloads its firmware, allowing for a complete reset of the expansion subsystem without having to power cycle the CPU.

Above: a snippet of a trace captured by the scope when probing a full-speed USB data line.

xobs also wrote a wonderful demo program in Qt for the oscilloscope, and through this we were able to do some preliminary performance characterization. The rise-time performance of the probe is everything I had hoped for, and the very long capture buffer provided by the FPGA’s DDR3 memory enables a new dimension of deep signal analysis. This, backed with Novena’s horsepower, tight integration with Linux and a hackable architecture makes for a compelling – and portable – signal analysis solution for field work.

If the prospect of a a hackable oscilloscope excites you as much as it does us, please consider backing our crowdfunding campaign for Novena and spreading the word to your friends; there’s only a few days left. Developing complex hardware and software systems isn’t cheap, and your support will enable us to focus on bringing more products like this to market.

Novena’s Hackable Bezel

May 3rd, 2014

When designing Novena, I had to balance budget against hackability. Plastic parts are cheap to produce, but the tools to mold them are very expensive and difficult to modify. Injection mold tooling cost for a conventional clamshell (two-body) laptop runs upwards of $250,000. In contrast, Novena’s single body design has a much lower tooling cost, making it feasible to amortize tooling costs over a smaller volume.

The decision to use flat sheet aluminum for the LCD bezel was also driven in part to reduce tooling costs. Production processing for aluminum can be done using CNC, virtually eliminating up-front tooling costs. Furthermore, aluminum has great hack value, as it can be cut, drilled, tapped, and bent with entry-level tools. This workability means end users can easily add connectors, buttons, sensors, and indicators to the LCD bezel. Users can even design in a custom LCD panel, since there’s almost no setup cost for machining aluminum.

One of my first mods to the bezel is a set of 3D-printed retainers, custom designed to work with my preferred keyboard. The retainers screw into a set of tapped M2.5 mounting holes around the periphery of the LCD.

The idea is that the retainers hold my keyboard against the LCD bezel when transporting the laptop, protecting the LCD from impact damage while making it a little more convenient for travel.

Such an easily customizable bezel means a limitless combination of keyboards and LCDs can be supported without requiring expensive modifications to injection molding tools.

The flat design also means it’s easy to laser-cut a bezel using other materials. Here’s an example made out of clear acrylic. The acrylic version looks quite pretty, although as a material acrylic is much softer and less durable than aluminum.

I also added a notch on the bottom part of the bezel to accommodate breakout boards plugged into the FPGA expansion connector.

The low up-front cost to modify and customize the bezel enables experimentation and serendipitous hacks. I’m looking forward to seeing what other Novena users do with their bezels!

Circuit Stickers Manufacturing Retrospective: From Campaign to First Shipment

April 29th, 2014

Last December, Jie Qi and I launched a crowdfunding campaign to bring circuit stickers under the brand name of “chibitronics” to the world.

Our original timeline stated we would have orders shipped to Crowd Supply for fulfillment by May 2014. We’re really pleased that we were able to meet our goal, right on time, with the first shipment of over a thousand starter kits leaving the factory last week. 62 cartons of goods have cleared export in Hong Kong airport, and a second round of boxes are due to leave our factory around May 5, meaning we’ve got a really good chance of delivering product to backers by Mid-May.

Above: 62 cartons containing over a thousand chibitronics starter kits waiting for pickup.

Why On-Time Delivery Is So Important
A personal challenge of mine was to take our delivery commitment to backers very seriously. I’ve seen too many under-performing crowdfunding campaigns; I’m deeply concerned that crowdfunding for hardware is becoming synonymous with scams and spams. Kickstarter and Indiegogo have been plagued by non-delivery and scams, and their blithe caveat emptor attitude around campaigns is a reflection of an entrenched conflict of interest between consumers and crowdfunding websites: “hey, thanks for the nickel, but what happened to your dollar is your problem”.

I’m honestly worried that crowdfunding will get such a bad reputation that it won’t be a viable platform for well-intentioned entrepreneurs and innovators in a few years.

I made the contentious choice to go with Crowd Supply in part because they show more savvy around vetting hardware products, and their service offering to campaigns — such as fulfillment, tier-one customer support, post-campaign pre-order support, and rolling delivery dates based on demand vs. capacity — is a boon for hardware upstarts. Getting fulfillment, customer support and an ongoing e-commerce site as part of the package essentially saves me one headcount, and when your company consists of just two or three people that’s a big deal.

Crowd Supply doesn’t have the same media footprint or brand power that Kickstarter has, which means it is harder to do a big raise with them, but at the end of the day I feel it’s very important to establish an example of sustainable crowdfunding practices that is better for both the entrepreneur and the consumer. It’s not just about a money grab today: it’s about building a brand and reputation that can be trusted for years to come.

Bottom line is, if I can’t prove to current and future backers that I can deliver on-time, I stand to lose a valuable platform for launching my future products.

On-Time Delivery Was not Easy
We did not deliver chibitronics on time because we had it easy. When drawing up the original campaign timeline, I had a min/max bounds on delivery time spanning from just after Chinese New Year (February) to around April. I added one month beyond the max just to be safe. We ended up using every last bit of padding in the schedule.

I made a lot of mistakes along the way, and through a combination of hard work, luck, planning, and strong factory relationships, we were able to battle through many hardships. Here’s a few examples of lessons learned.

A simple request for one is not necessarily a simple request for another. Included with every starter kit is a fantastic book (free to download) written by Jie Qi which serves as a step-by-step, self-instruction guide to designing with circuit stickers. The book is unusual because you’re meant to paste electronic circuits into it. We had to customize several aspects of the printing, from the paper thickness (to get the right light diffusion) to the binding (for a better circuit crafting experience) to the little pocket in the back (to hold swatches of Z-tape and Linqstat material). Most of these requests were relatively easy to accommodate, but one in particular threw the printer for a loop. We needed the metal spiral binding of the book to be non-conductive, so if someone accidentally laid copper tape on the binding it wouldn’t cause a short circuit.

Below is an example of how a circuit looks in the book — in this case, the DIY pressure sensor tutorial (click on image for a larger version).

Checking for conductivity of a wire seems like a simple enough request for someone who designs circuits for a living, but for a book printer, it’s extremely weird. No part of traditional book printing or binding requires such knowledge. Because of this, the original response from the printer is “we can’t guarantee anything about the conductivity of the binding wire”, and sure enough, the first sample was non-conductive, but the second was conductive and they could not explain why. This is where face to face meetings are invaluable. Instead of yelling at them over email, we arranged a meeting with the vendor during one of my monthly trips to Shenzhen. We had a productive discussion about their concerns, and at the conclusion of the meeting we ordered them a $5 multimeter in exchange for a guarantee of a non-conductive book spine. In the end, the vendor was simply unwilling to guarantee something for which he had no quality control procedure — an extremely reasonable position — and we just had to educate the vendor on how to use a multimeter.

To wit, this unusual non-conductivity requirement did extend our lead time by several days and added a few cents to the cost of the book, but overall, I’m willing to accept that compromise.

Never skip a checkplot. I alluded to this poignant lesson with the following tweet:


The pad shapes for chibitronics are complex polyline geometries, which aren’t handled so gracefully by Altium. One problem I’ve discovered the hard way is the soldermask layer occasionally disappears for pads with complex geometry. One version of the file will have a soldermask opening, and in the next save checkpoint, it’s gone. This sort of bug is rare, but it does happen. Normally I do a gerber re-import check with a third-party tool, but since this was a re-order of an existing design that worked before, and I was in a rush, I skipped the check. Result? thousands of dollars of PCBs scrapped, four weeks gone from the schedule. Ouch.

Good thing I padded my delivery dates, and good thing I keep a bottle of fine scotch on hand to help bitter reminders of what happens when I get complacent go down a little bit easier.

If something can fit in a right and a wrong way, the wrong way will happen. I’m paranoid about this problem — I’ve been burned by it many times before. The effects sticker sheet is a prime example of this problem waiting to happen. It is an array of four otherwise identical stickers, except for the LED flashing pattern they output. The LED flashing pattern is controlled by software, and trying to manage four separate firmware files and get them all loaded into the right spot in a tester is a nightmare waiting to happen. So, I designed the stickers to all use exactly the same firmware; their behaviors set by the value of a single external resistor.

So the logic goes: if all the stickers have the same firmware, it’s impossible to have a “wrong way” to program the stickers. Right?

Unfortunately, I also designed the master PCB panels so they were perfectly symmetric. You can load the panels into the assembly robot rotated by pi radians and the assembly program runs flawlessly — except that the resistors which set the firmware behavior are populated in reverse order from the silkscreen labels. Despite having fiducial holes and text on the PCBs in both Chinese and English that are uniquely orienting, this problem actually happened. The first samples of the effects stickers were “blinking” where it said “heartbeat”, “fading” where it said “twinkle”, and vice-versa.

Fortunately, the factory very consistently loaded the boards in backwards, which is the best case for a problem like this. I rushed a firmware patch (which is in itself a risky thing to do) that reversed the interpretation of the resistor values, and had a new set of samples fedexed to me in Singapore for sanity checking. We also built a secondary test jig to add a manual double-check for correct flashing behavior on the line in China. Although, in making that additional test, we were confronted with another common problem –

Some things just don’t translate well into Chinese. When coming up with instructions to describe the difference between “fading” (a slow blinking pattern) and “twinkling” (a flickering pattern), it turns out that the Chinese translation for “blink” and “twinkle” are similar. Twinkle translates to 闪烁 (“flickering, twinkling”) or 闪耀 (to glint, to glitter, to sparkle), whereas blink translates to 闪闪 (“flickering, sparkling, glittering”) or 闪亮 (“brilliant, shiny, to glisten, to twinkle”). I always dread making up subjective descriptions for test operators in Chinese, which is part of the reason we try to automate as many tests as possible. As one of my Chinese friends once quipped, Mandarin is a wonderful language for poetry and arts, but difficult for precise technical communications.

Above is an example of the effects stickers in action. How does one come up with a bulletproof, cross-cultural explanation of the difference between fading (on the left) and twinkling (on the right), using only simple terms anyone can understand, e.g. avoiding technical terms such as random, frequency, hertz, periodic, etc.

After viewing the video, our factory recommended to use “渐变” (gradual change) for fade and “闪烁” (flickering, twinkling) for twinkle. I’m not yet convinced this is a bulletproof description, but it’s superior to any translation I could come up with.

Funny enough, it was also a challenge for Jie and I to agree upon what a “twinkle” effect should look like. We had several long conversations on the topic, followed up by demo videos to clarify the desired effect. The implementation was basically tweaking code until it “looked about right” — Jie described our first iteration of the effect as “closer to a lightning storm than twinkling”. Given the difficulty we had describing the effect to each other, it’s no surprise I’m running into challenges accurately describing the effect in Chinese.

Eliminate single points of failure. When we built test jigs, we built two copies of each, even though throughput requirements demanded just one. Why? Just in case one failed. And guess what, one of them failed, for reasons as of yet unknown. Thank goodness we built two copies, or I’d be in China right now trying to diagnose why our sole test jig isn’t working.

Sometimes last minute changes are worth it. About six weeks ago, Jie suggested that we should include a stencil with the sensor/microcontroller kits. She reasoned that it can be difficult to lay out the copper tape patterns for complex stickers, such as the microcontroller (featuring seven pads), without a drawing of the contact patterns. I originally resisted the idea — we were just weeks away from finalizing the order, and I didn’t want to delay shipment on account of something we didn’t originally promise. As Jie is discovering, I can be very temperamental, especially when it comes to things that can cause schedule slips (sorry Jie, thanks for bearing with me!). However, her arguments were sound and so I instructed our factory to search for a stencil vendor. Two weeks passed and we couldn’t find anyone willing to take the job, but our factory’s sourcing department wasn’t going to give up so easily. Eventually, they found one vendor who had enough material in stock to tool up a die cutter and turn a couple thousand stencils within two weeks — just barely in time to meet the schedule.

When I got samples of the sensor/micro kit with the stencils, I gave them a whirl, and Jie was absolutely right about the utility of the stencils. The user experience is vastly improved when you have a template to work from, particularly for the microcontroller sticker with seven closely spaced pads. And so, even though it wasn’t promised as part of the original campaign, all backers who ordered the sensor/micro kit are getting a free stencil to help with laying out their designs.

Chinese New Year has a big impact the supply chain. Even though Chinese New Year (CNY) is a 2-week holiday, our initial schedule essentially wrote off the month of February. Reality matched this expectation, but I thought it’d be helpful to share an anecdote on exactly how CNY ended up impacting this project. We had a draft manuscript of our book in January, but I couldn’t get a complete sample until March. It’s not because the printer was off work for a month straight — their holiday, like everyone else’s, was about two weeks long. However, the paper vendor started its holiday about 10 days before the printer, and the binding vendor ended its holiday about 10 days after the printer. So even though each vendor took two weeks off, the net supply chain for printing a custom book was out for holiday for around 24 days — effectively the entire month of February. The staggered observance of CNY is necessary because of the sheer magnitude of human migration that accompanies the holiday.

Shipping is expensive, and difficult. When I ran the initial numbers on shipping, one thing I realized is we weren’t selling circuit stickers — at least by volume and weight, our principle product is printed paper (the book). So, to optimize logistics cost, I was pushing to ship starter kits (which contain a book) and additional stand-alone book orders by ocean, rather than air.

We actually had starter kits and books ready to go almost four weeks ago, but we just couldn’t get a reasonable quotation for the cost of shipping them by ocean. We spent almost three weeks haggling and quoting with ocean freight companies, and in the end, their price was basically the same as going by air, but would take three weeks longer and incurred more risk. It turns out that freight cost is a minor component of going by ocean, and you get killed by a multitude of surcharges, from paying the longshoreman to paying all the intermediate warehouses and brokers that handle your goods at the dock. All these fixed costs add up, such that even though we were shipping over 60 cartons of goods, air shipping was still a cost-effective option. To wit, a Maersk 40′ sea container will fit over 1250 cartons each containing 40 starter kits, so we’re still an order of magnitude away from being able to efficiently utilize ocean freight.

We’re not out of the Woods Yet. However excited I am about this milestone, I have to remind myself not to count my chickens before they hatch. Problems ranging from a routine screw-up by UPS to a tragic aviation accident to a logistics problem at Crowd Supply’s fulfillment depot to a customs problem could stymie an on-time delivery.

But, at the very least, at this point we can say we’ve done everything reasonably within our power to deliver on-time.

We are looking forward to hearing our backer’s feedback on chibitronics. If you are curious and want to join in on the fun, the Crowd Supply site is taking orders, and Jie and I will be at Maker Faire Bay Area 2014, in the Expo hall, teaching free workshops on how to learn and play with circuit stickers. We’re looking forward to meeting you!