Modding devices with eMMC flash

A lot of networking device are coming out with eMMC flash storage instead of SPI or even NAND flash, making it difficult to read and write flash out of band. What if we can use pluggable flash modules instead of soldered BGA eMMC flash? It turns out that with some soldering skills you can convert soldered eMMC flash to pluggable module that can be read and written outside of device. Good use cases for this are replacing bad flash or expanding storage on out of warranty Edgerouter 4/6P/12 routers, or accessing filesystems of newer routers used by ISPs.

Warning: This guide describes process that requires specific skills and safety measures. I am not responsible if you brick your device, set your house on fire or kill your neighbors dog while following this guide.

Before attempting this, I have never soldered BGA devices in my life. This takes some care and patience, but if someone like me with no experience can do this by following Youtube videos, it's likely others may also be successful in it. If you want to mod your device and don't feel comfortable doing it yourself, your local cell phone repair store may have the right skill set and equipment to do this for you.

It goes without saying that you should not be modding anything owned by your ISP.

I got inspiration for this from ODROID project, which utilizes pluggable modules for eMMC flash.
The pinouts of connectors described in this guide are fully compatible with ODROID - you should be able to use ODROID compatible flash readers and use ODROID compatible eMMC flash modules with your modded devices (as long as the devices can recognize flash modules).

In this mode we are desoldering existing eMMC flash from device being modded, and replacing it with an interposer - a special PCB with eMMC BGA footprint on one side and ODROID-compatible flash connector on the other side.

I have designed an eMMC interposer using eMMC IC datasheets and ODROID adapter schematics and designed an eMMC module PCB in eMMC 11.5x13mm footprint.
This is the second time I am designing my own PCBs, so the board layouts may look amateurish. But they do work. Data traces lengths are tuned to +/-1 mil.

A picture is worth a thousand words:

What you see on the pictures above is a consumer router, same model that is used by some ISPs that I purchased for experiments. The router's eMMC flash has been desoldered, re-balled and soldered to an ODROID-compatible flash module PCB. An interposer PCB has been re-balled (to behave like an eMMC module) and soldered in place of eMMC.

PCBs for both interposer and eMMC-compatible module can be ordered from OSHPARK for less than $4 for 3 boards.

eMMC interposer

ODROID-compatible eMMC module

The boards are using smaller tolerances than allowed by OSHPARK, so while I've had success with the designs above, it is possible that the boards may not work if the drill is shifted too much from center of vias. Use microscope or zoom in on a picture taken from your phone to inspect the vias. You want the drill hole to be as close to the center of vias as possible, or at least within the surface of the via - not protruding from it's footprint. If the drill hole is too off-center, try ordering again - a different batch may be centered better.

Besides the boards you will need some tools and supplies.

For eMMC interposer you need:

• GB042-34P-H10 connector
• 0.3mm solder balls. Low-temperature solder balls are easier to work with. I've used Sn/Pb solder balls
  

For eMMC module you need more parts:

• GB042-34S-H10 connector
• 0.3mm solder balls to re-ball your eMMC IC (unless you are using a new one factory-reballed)
• Some capacitors and resistors in 0603 SMD form factor listed at OSHPARK board page. 

When ordering parts and PCBs, unless you are very good at SMD soldering, it's best to order more than you need in case you you don't succeed from first attempt.

You would also need some generic tools and supplies:

• Soldering iron with temperature control
• Something that can be used for de-soldering an re-soldering BGA like hot plate, skillet, hot air station, PCB reflow oven, IR rework station. You can find techniques for soldering BGA using the described tools in Youtube. I have used a DIY reflow oven converted from a toaster and a hot air gun (with temperature control).
  
• Digital Multimeter, preferably with thermocouple that can be used to measure board temperature
• Solder wick
• Kapton tape (1 mil thickness)
• Rosin flux
• Isopropyl alcohol for cleaning contacts and removing flux
• Solder paste and/or solder. I like using Sn/Pb, but other low-temperature solders should work too.
• HVAC aluminum tape to cover plastic parts of the board when re-flowing BGA.
• Tweezers
• IC suction tool (not necessary, but helps to cleanly desolder eMMC IC)
• LCD microscope is not strictly necessary if your vision is good, but is very handy to have
• BGA reballing station is not necessary, but makes working with small parts a lot more comfortable. I've used this kind.
• Having eMMC stencils (and PCB stencils) makes it a lot easier to accomplish the steps in this guide. I've used my own eMMC stencil, but generic ones should work too. Below are links to my stencils. I've used oshstencils.com to order the stencils in 4 mil stainless steel. Below are the gerber files for stencils used in this guide:
• eMMC BGA
• Interposer
• eMMC Module

Now let's get to the actual process.

The first step (after carefully disassembling your device) is desoldering eMMC IC from your device. eMMC flash chip is likely soldered with lead-free solder with a melting point over 217°C/422°F. Heat up the eMMC IC, but don't overheat it. Suction tool helps to lift the IC without damage, but you can also use tweezers to carefully flip the chip when solder under the chip melts. To avoid overheating the board and IC you could tape thermal probe to the board next to eMMC IC and monitor the temperature while de-soldering. It is also helpful to preheat the whole board to avoid thermal stress of heating just one part of the board.

I have used a DIY reflow oven set for leaded solder thermal envelope to heat up the whole board to the top of leaded thermal envelope, then removed the board from oven and heated the eMMC some more with hot air until I was able to remove it with suction tool.

I have also wrapped all the plastic parts with HVAC aluminum tape to avoid melting them

The next step is removing lead-free solder from the board and eMMC IC. Use solder wick and either soldering iron or hot air, being careful not to blow away SMD devices next to eMMC landing pad. Use kapton tape to cover SMDs if using hot air. I found that the easiest way to clean the pads is to use a dab of solder paste on solder wick and sliding wick over the pads with soldering iron. Not all the pads of eMMC IC are used. Most of the pads are there just for structural purposes - to hold the IC in place. If some of non-connected pads delaminate when cleaning, it is not the end of the world. If using solder instead of solder paste, use rosin flux. I've tried to be extra careful, but still got a few pads lifted on this board. Likely they are not connected to any traces. After cleaning the surface of the board and surface of eMMC chip need to be completely flat. Use rubbing alcohol to clean flux. 

Now let's attach some solder balls to interposer. Cover interposer with a really thin layer of sticky rosin flux and attach the balls. Using stencil makes this quick and easy. Some people use direct heat stencils with solder paste. Whatever works best for you. Once the solder balls are stuck to flux, heat the board on hot plate, skillet or reflow oven to melt them.

Boards come from OSH Park with tabs used for panelling. Tabs can be easily filed off.

Repeat the process with eMMC IC. Solder the connector to interposer. You could use the stencil and solder paste, or you can hand solder the connector. Check out Youtube videos on techniques to solder 0.4mm pitch connectors. Hand soldering requires a special technique and some skill to avoid bridged contacts.

Before soldering eMMC Interposer to modded device I've decided to solder it to an eMMC to MicroSD card adapter to check if I can read ODROID eMMC modules using this interposer.
I've ordered adapter from OSHPARK using gerbers from here: https://github.com/voltlog/emmc-wfbga153-microsd but factory-made adapters from Ebay or other sites would work too. If ordering eMMC to SD adapter from OSH Park using gerbers from GitHub, use 2oz copper boards - they are thinner and closely match thickness of SDXC cards.

Important note: eMMC Interposer is about five times heavier than regular eMMC IC. Use two layers of 1 mil thick kapton tape as standoff when reflowing eMMC Interposer to avoid solder balls collapse causing bridging and electrical shorts. Cover PCB with a very thin layer of rosin flux and attach kapton tape.

The step above is not strictly necessary. You can use off the shelf ODROID adapter to read flash, but it provides confidence in your soldering skills.

Now that we are confident that our eMMC Interposer works, let's solder one to our device. Again, use two layers of 1 mil thick kapton tape to avoid solder balls from collapsing under the weight of interposer and use a very thin layer of rosin flux. If you use too much flux it may boil and shift the interposer.

I've used reflow oven at leaded solder profile setting. It is not hot enough to melt the solder for the rest of components on this board since it uses lead-free solder, which requires higher temperature to melt than leaded solder.

Now let's assemble the ODROID-compatible eMMC module with eMMC IC we have desoldered from the device. You can use off the shelf ODROID-compatible modules if our device can recognize flash. You would still need the means to read existing flash and copy it to replacement module.

Start with soldering the passive SMD components, then flip the module PCB and solder eMMC IC (that has been previously reballed) to module.

Here you can see comparison between the factory ODROID eMMC module and the one I've soldered myself.

Let's plug the original flash on top of the new module into the device and check if it works.

It does!

Let's try with a different router I've purchased for experiments. Before we tear it apart, let's check that it actually works.

It does.

Heatsink off! Now let's get EM shield off.

I've used the same process to desolder IC as with the blue board router above. I've heated the whole board to leaded solder melting point and used hot air gun on eMMC IC to get it to lead-free solder melting point. I am using PCB to hide serial on label.

Remove lead-free solder and clean eMMC IC

Same with the board. Either the board FR4 rating is higher than for the blue board, or I just got lucky enough not to lift any pads on this device.

Time to solder interposer. Same technique with two layers of kapton tape for standoffs.

This is how it looks after reflowing:

Let's now test with original flash and a copy.

Success!

Time for EM shield and heatsink surgery using dremel.

Can't say it looks good, but it works.

Before attaching eMMC to SD adapter to a PC, make sure to disable automount. Don't try to manipulate block devices that are mounted - you are likely to corrupt the date.

I've used the following commands to make backup copy of contents of eMMC:

sudo dd if=/dev/mmcblk0boot0 of=mmcblk0boot0.img bs=1024k status=progress

sudo dd if=/dev/mmcblk0boot1 of=mmcblk0boot1.img bs=1024k status=progress

sudo dd if=/dev/mmcblk0 of=mmcblk0.img bs=1024k status=progress

To flash this backup to a new eMMC I've used the following:

sudo mmc bootbus set single_backward x1 x8 /dev/mmcblk0

sudo mmc bootpart enable 1 1 /dev/mmcblk0

sudo mmc enh_area set -y 0 1908736 /dev/mmcblk0

(Remove and plug back eMMC)

sudo su

echo 0 > /sys/block/mmcblk0boot0/force_ro

echo 0 > /sys/block/mmcblk0boot1/force_ro

exit

sudo dd of=/dev/mmcblk0boot0 if=mmcblk0boot0.img bs=4k status=progress

sudo dd of=/dev/mmcblk0boot1 if=mmcblk0boot1.img bs=4k status=progress

sudo dd of=/dev/mmcblk0 if=mmcblk0.img bs=4k status=progress

Disabling ChromeOS automatic updates

I really like ChromeOS. Unfortunately Google is not very good with QA and new updates to my ChromeOS device make it completely unusable to me. It keeps getting worse with more functionality getting broken. I've had to roll back using recovery media, but the device keeps updating itself automatically.

To disable ChromeOS updates, I have created custom DNS 'A' records on my router:

tools.google.com 127.0.0.1
dl.google.com 127.0.0.1  

I configured DHCP scope for my ChromeOS device to use my router as DHCP server. 
Now my ChromeOS device reports that ChromeOS is up to date and never tries to upgrade itself.
It's a shame that I have to disable security updates to avoid making my device unusable.

Statement on fraudulent DMCA complaints

It came to my attention that someone pretending to be me is sending DMCA takedown requests for several GitHub repositories, claiming the repositories are infringing on my copyright for mfg_dat_decode, also known as 802.1x Credential Extraction Tool. 

I hereby affirm that as a copyright holder for the abovementioned tool, I have never sent DMCA takedown requests for any GitHub repositories.

I do authorize the distribution and use of mfg_dat_decode in its binary unmodified form in any open source projects for non-commercial purposes. Any commercial use, including use in any legal proceedings requires explicit license. Decompiling, disassembling or any other reverse engineering or modification of this tool for any commercial use is strictly prohibited.

Stadia Controller failing to connect to Cisco Wireless APs

When I received my Stadia Controller Founder's Edition, I've noticed that I cannot connect to any SSID on my Cisco Wireless Access points. I've tried moving my 802.11ac Wave 2 APs from WLC to Mobility Express and back, creating new WPA3 and WPA2 SSIDs and still cannot get the Stadia Controller to connect. I've even taken out my old 3600 AP, reflashed it with Autonomous image and tried to set up a plain vanilla WPA2 SSID on it. Stadia Controller still would not connect. That was very odd, considering I have dozens of wireless clients of various age and processing power, including Google/Nest devices that connect with no issues.

My next step was setting up a test SSID on Meraki MR24 AP reflashed with lates trunk image of OpenWRT. Stadia Controller connected to my test WPA3 SSID right away. Test with WPA2 was also successful. But that was not an acceptable workaround for me, since I don't want to keep another AP up and running just for the Stadia Controller, so I've decided to take time over the weekend to spin up Kali Linux on one of my laptops, to do some packet captures in monitor mode and compare the attributes of beacon frames.

Here is what I have discovered:

Starting with Cisco WLC version 8.3.1020, IEEE 802.11v and IEEE 802.11k are enabled for new SSIDs by default.
According to Cisco, "Wireless network management (802.11v) strives to improve the quality of the end-user experience by enabling information exchange. Basic Service Set (BSS) transition management is a vital aspect of 802.11v by which devices can solicit advice from the WLAN as well as receive unsolicited advice from the WLAN about which access point they should associate to. The decision to associate or not would ultimately depend on the device. Additionally, 802.11v includes multiple extensions that enable the client to sleep for a longer duration and thereby save battery life."
Unfortunately, it seems that this undoubtedly helpful feature is not compatible with Google Stadia Controller.

Once I've turned off 802.11v on my SSIDs, Stadia Controller was immediately able to connect.

I don't know if Stadia Controller is using wpa_supplicant, or some in-house WPA Supplicant implementation, but whatever they use, it seems that the implementation is buggy.
Hopefully it is fixed in future, but meanwhile, if you are experiencing issues connecting your Stadia Controller to WLC-managed enterprise APs, try disabling 802.11v. Here is a good article describing how to check SSID for 802.11v support and how to enable/disable it for SSID on a Cisco WLC.
In Wireshark you would want to look at beacon frames, IEEE 802.11 wireless LAN, Tagged Parameters, Extended Capabilities, Octet 3.

BMW NBT and NBT Evo Firmware Unpacker

It has been five years since I have released the first build of NBT firmware unpacker.

A few years ago I've had to stop the distribution of unpacker because the misuse of unpacker was disrupting the business of BMW retrofit shops. With 2018 public release of NBT and EVO vulnerabilities that have been known to the BMW retrofit scene since at least fall of 2014, and subsequent patching of vulnerabilities by BMW, I no longer consider distribution of the unpacker as harmful.

To commemorate the five years of NBT unpacker, I am releasing an updated and refactored version with the following new features:

• Search for supported unpackable firmware files within specified folder
• Searching for specific file within firmware across multiple firmware files in specified folder
• Better handling of special cases like ASD or EQU files.

Download nbt_unpack release 1.0.0 here: Win32

Configuring 802.1x authentication using wpa_supplicant on Ubiquiti Edgerouter

This guide describes steps required to configure 802.1x wired authentication using wpa_supplicant on Cavium-based Ubiquiti Edgerouter devices running EdgeOS 2.0. Mediatek (Edgerouter-X) procedure is similar, however I do not have such device and could not recompile wpa_supplicant for it.
Copy EAP-TLS certificates and private key in PEM format to /config/auth/
Run sudo chmod -R 0600 /config/auth to secure the credentials.
Copy wpa_supplicant.conf to /config/
If you are using wpa_supplicant.conf generated by a tool in the previous post, modify certificate and key paths to point to /config/auth/

Assuming wpa_supplicant.conf resides in /config/, the router has Internet connectivity and Internet interface that requires wpa_supplicant is eth0, run the following commands in console:

#Remove 1.x repository, add 2.0 Debian repository and install prerequisites
configure
delete system package repository wheezy
set system package repository stretch components 'main contrib'
set system package repository stretch distribution stretch
set system package repository stretch password ''
set system package repository stretch url 'http://http.us.debian.org/debian'
set system package repository stretch username ''
commit
save
exit
sudo apt-get update && sudo apt-get install libpcsclite1
#Download backported Debian Buster wpa_supplicant and install it
curl https://community.ubnt.com/ubnt/attachments/ubnt/EdgeMAX/235525/1/wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz -o /tmp/wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz
cd /tmp/
tar -xvf ./wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz
sudo dpkg -i /tmp/wpasupplicant_2.6-21~bpo9+1_mips.deb
#Create symbolic link to wpa_supplicant.conf for eth0
sudo ln -s /config/wpa_supplicant.conf /etc/wpa_supplicant/wpa_supplicant-wired-eth0.conf
#Disable dbus service and enable wired wpa_supplicant for eth0
sudo systemctl disable wpa_supplicant.service
sudo systemctl enable wpa_supplicant-wired@eth0.service
#Save DEB packages for future use, allow /etc/ubnt/ubnt-rcS/ubnt-rcS.sh to install them
sudo mkdir -p /config/data/firstboot/install-packages &&  cd /config/data/firstboot/install-packages
sudo apt-get download libpcsclite1sudo curl https://community.ubnt.com/ubnt/attachments/ubnt/EdgeMAX/235525/1/wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz -o /config/data/firstboot/install-packages/wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz
sudo tar -xvf ./wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz
sudo rm /config/data/firstboot/install-packages/wpasupplicant_2.6-21~bpo9+1_mips.deb.tar.gz
#Recover configuration on EdgeOS upgrade
sudo mkdir -p /config/scripts/firstboot.d/
echo '#!/usr/bin/env bash' | sudo tee -a /config/scripts/firstboot.d/8021x-eth0.sh
echo 'ln -s /config/wpa_supplicant.conf /etc/wpa_supplicant/wpa_supplicant-wired-eth0.conf' | sudo tee -a /config/scripts/firstboot.d/8021x-eth0.sh
echo 'systemctl stop wpa_supplicant.service' | sudo tee -a /config/scripts/firstboot.d/8021x-eth0.sh
echo 'systemctl disable wpa_supplicant.service' | sudo tee -a /config/scripts/firstboot.d/8021x-eth0.sh
echo 'systemctl enable wpa_supplicant-wired@eth0.service' | sudo tee -a /config/scripts/firstboot.d/8021x-eth0.sh
echo 'systemctl start wpa_supplicant-wired@eth0.service' | sudo tee -a /config/scripts/firstboot.d/8021x-eth0.sh
sudo chmod 0744 /config/scripts/firstboot.d/8021x-eth0.sh
reboot