Supermind Search Consulting Blog 
Solr - Elasticsearch - Big Data

If you only need to route traffic on Android through a ssh tunnel (not proxy), just use http://code.google.com/p/sshtunnel/
If all you need to do is to inspect network traffic, you can use Wireshark on Genymotion.

If however, you're on Genymotion and/or need to get Android traffic through a proxy, especially if you're trying to conduct MITM attacks or need to inspect network traffic, read-on. This article also assumes you're on either a Mac or Linux computer. Sorry, Windows users.

My requirements for this setup were:
1. inspect, save and playback network traffic
2. capture *all* incoming and outgoing TCP traffic from android devices, not just HTTP connections
3. write code to mangle and manipulate traffic as needed

I tried pretty much every technique I knew about and found on the internet, and found only *one* method consistently reliable and that worked on Genymotion: a combination of ssh, redsocks, iptables and a python proxy. To use this setup for an physical Android device, I connect it to the same wireless network, and then use ConnectBot to ssh-tunnel into the host computer.

Genymotion

First let's talk about getting this working on Genymotion.

Genymotion runs the Android emulator in a virtualbox instance. Contrary to what many report online, setting the proxy via Settings > Wifi > Manage really isn't enough. It doesn't force all network connections through a particular proxy. Interestingly, setting the proxy at host operating system level didn't work either (in Ubuntu 12.04 at least). Virtualbox seemed to be ignoring this setting and connecting directly. And running tsocks genymotion didn't work too. Neither did using connectbot or proxydroid with global proxy enabled.

Having said that, you should try the Wifi proxy approach first. It's way way easier and may be sufficient for your needs!

In the Genymotion Android emulator
Settings -> Wifi -> Long-presson active network
Select "Modify Network"
Select "Show Advanced Options"
Select "Proxy Settings -> Manual"
Hostname: 10.0.3.2
Port: 8888
Press Save

(this assumes Charles is listening in on port 8888). The 10.0.3.2 is Genymotion's special IP for the host IP. Now, if you use the Android browser, requests *should* be visible in Charles. If this is not happening, then something is wrong. If Android browser requests are visible in Charles but not the traffic you're interested in, then read on…

As this link explains really well, virtualbox operates at a lower level than socks, it's actually not possible to route virtualbox connections directly through a socks proxy. We'll get around this by using iptables and redsocks.

Our final application stack will look like this:

TCP requests from applications > iptables > redsocks > python proxy > ssh tunnel > internet

I'm almost positive it's possible to eliminate one or more steps with the right iptables wizardry, but I had already wasted so much time getting this working, I was just relieved to have a working solution.

First, setup your SSH tunnel with dynamic port forwarding (SOCKS forwarding) on port 7777. (I assume there is a SSH server you can tunnel into. If you don't have one, try the Amazon EC2 free-pricing tier)

ssh username@host -D 7777

Then setup iptables to route all TCP connections to port 5555. Remember: you have to do the SSH connect *before* changing iptables. Once the ssh connection is live, it stays alive (assuming you have keep-alive configured).

Put the following into a script called redsocks-iptables.sh

sudo iptables -t nat -N REDSOCKS
 
# Ignore LANs and some other reserved addresses.
sudo iptables -t nat -A REDSOCKS -d 0.0.0.0/8 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 10.0.0.0/8 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 127.0.0.0/8 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 169.254.0.0/16 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 172.16.0.0/12 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 192.168.0.0/16 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 224.0.0.0/4 -j RETURN
sudo iptables -t nat -A REDSOCKS -d 240.0.0.0/4 -j RETURN
 
# Anything else should be redirected to port 5555
sudo iptables -t nat -A REDSOCKS -p tcp -j REDIRECT --to-ports 5555
 
# Any tcp connection made by 'user' should be redirected, put your username here.
sudo iptables -t nat -A OUTPUT -p tcp -m owner --uid-owner user -j REDSOCKS

This iptables script basically redirects all tcp connections to port 5555.

And put this in redsocks-iptables-reset.sh

sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -F
sudo iptables -X

Use this to reset your iptables to the original state (assuming you don't already have any rules in there).

Now we use redsocks to route these tcp connections to a socks proxy. You may have to install some dependencies such as libevent.

git clone https://github.com/darkk/redsocks.git
cd redsocks
make

Save this as redsocks.conf

base{log_debug = on; log_info = on; log = "file:/tmp/reddi.log";
       daemon = on; redirector = iptables;}
       redsocks { 
       local_ip = 127.0.0.1; 
       local_port = 5555; 
       ip = 127.0.0.1;
       port = 6666; type = socks5; }

And run redsocks like so:

redsocks -c redsocks.conf

Now we need to setup the python proxy. This listens in on port 6666 (the port redsocks forwards requests to) and forwards requests out to 7777 (the ssh socks port). Download it from http://code.activestate.com/recipes/502293-hex-dump-port-forwarding-network-proxy-server/

Rename it to hexdump.py, and run it like so:

python ./hexdump.py 6666:localhost:7777

Now you're all setup. When you make *any* request from your Genymotion android device, it should appear as coming from your ssh server. And the hexdump of the data appears in the console from which you ran hexdump.py like so.

2014-11-16 01:04:51,340 INFO client 127.0.0.1:38340 -> server localhost:7777 (37 bytes)
-> 0000   17 03 01 00 20 EF F3 1C 57 4C 02 8C AA 56 13 53    .... ...WL...V.S
-> 0010   A7 7E E0 7B B9 55 CC 32 6B 5D 42 AC B8 01 52 6B    .~.{.U.2k]B...Rk
-> 0020   2D 80 2A 44 08                                     -.*D.

Physical Android devices

To inspect traffic from a physical Android device, I had to go 2 steps further:

1. Setup ssh server locally (where redsocks is running) NOT the destination ssh server
2. Install connectbot and route all network through host (remember to enable the Global proxy option)

Yes, unfortunately, root is required on the device to use the 'Global proxy' option. I read someplace about a non-root workaround using an app called Drony which uses a Android VPN to route traffic to a proxy of your choice, but I didn't get this to work. Maybe you can.

Manipulating traffic

The Python script is pretty easy to decipher. The 2 important functions are:

  def dataReceived(self, recvd):
    logger.info("client %s -> server %s (%d bytes)\n%s" % (
      self.clientName, self.serverName, len(recvd), hexdump('->', recvd)))
    if hasattr(self, "dest"):
      self.dest.write(recvd)
    else:
      logger.debug("caching data until remote connection is open")
      self.clientFactory.writeCache.append(recvd)
 
  def write(self, buf):
    logger.info("client %s <= server %s (%d bytes)\n%s" % (
      self.clientName, self.serverName, len(buf), hexdump('<=', buf)))
    self.transport.write(buf)

Here you can intercept the data, inspect it, write it out to file, or modify it as you wish.

Notes

The really nice thing about using the python proxy is how easy it is to write data to file and to write rules to manipulate both incoming and outgoing traffic. Beats wireshark any day of the week. As I'm a little more familiar with Java, I initially got this working using Netty example's HexDumpProxy, but manipulating data in Netty was a gigantic PITA, not to mention the painful edit-compile-run cycle.

The major downside of this approach is not being able to (easily) see which server the data you're manipulating is destined for/coming from: the client and servers are both proxies. There's probably a way around this. If you figure it out, let me know!

If you need to decrypt and inspect SSL connections (I didn't), you can add Charles into the mix, after the python proxy and before the SSH tunnel. Just set the python proxy to out-go to the same port the Charles socks proxy is listening on, and set Charles' upstream socks proxy to the SSH tunnel port (in our case 7777). You'll find this setting in Charles under Proxy > External Proxy Settings

If at times one of the links in the chain breaks down, like the ssh connection dying, you may need to kill redsocks, reset iptables and start over. I didn't have to do this too often.

8 Comments »

Why bother upgrading? 2 simple reasons: USB and wifi tethering.

Instructions courtesy of my friend Jack:

Step 1) Do a complete backup of your SD card data (just in case)

Step 2) Root your phone
– Go to http://forum.xda-developers.com/showthread.php?t=742228 and follow the instructions

Step 3) Do a Nand backup
– Make sure you have >=500Mb free on your SD card
– With phone off, hold Power + Volume Up to boot into recovery
– Choose the Nand backup option
– Copy the nandroid folder from your SD card to your computer

Step 4) Load the ROM of your choice
– Download a ROM (I recommend http://forum.xda-developers.com/showthread.php?t=745603)
– Follow directions from the ROM page, which generally will include
– Put the ROM in the root dir of your SD card
– Reboot phone into recovery mode (like in step 3)
– Wipe data / factory reset, and wipe Dalvik-cache
– Flash zip from SD card
– Wait for a long time

If you're having difficulty getting to the recovery console like me, try Volume Down + Power instead.

2 Comments »

Found this out through a friend:

1. Hit *#*#4636#*#*
2. Phone Info
3. Select WCDMA in Preferred Network.

More information on preferred networks available here: http://www.google.com/support/forum/p/android/thread?tid=6a327a95211ac789&hl=en

WCDMA preferred – The GSM phone is capable of using both 2G and 3G data communication and when signal strength is low 3G is favored more.
GSM only – The GSM phone is capable of using only 2G data communication. When the 2G signal is too low you get nothing at all.
WCDMA only – The GSM phone is capable of using only 3G data communication. When the 3G signal is too low you get nothing at all.
GSM auto (PRL) – The GSM phone is capable of using both 2G and 3G data communication and when signal strength is low 2G is favored more. This one is a bit confusing to me since PRL is associated mostly with CDMA technology and not GSM technology.
CDMA auto (PRL) – The CDMA phone is capable of using both 2G and 3G data communication and when signal strength is low 2G is favored more.
CDMA only – The CDMA phone is capable of using only 2G data communication. When the 2G signal is too low you get nothing at all.
EvDo only – The CDMA phone is capable of using only 3G data communication. When the 3G signal is too low you get nothing at all.
GSM/CDMA auto (PRL) – Some phones are equipped with both GSM and CDMA capabilities. This setting appears to just have the phone attempt to stay connected to the data communication type that works the best. (Maybe the Samsung Galaxy S will take advantage of this???)
Unknown – If none of the above fit or the phone is acting weird as far as connecting to the carrier, you will see your preferred network type is set to this

1 Comment »

Android's adb bridge, when performing port forwarding to the Android device, only accepts connections from localhost. This is most probably for security reasons, but is a nuisance when you want to be able to access the net via your Android device from >1 computer over the network.

The solution: http://rxwen.blogspot.com/2009/11/adb-for-remote-connections.html

Its a drop-in replacement for your ANDROID_HOME/tools/adb file.

1 Comment »

After updating to Android 2.1, Windows XP no longer recognized my HTC Droid Eris.

The fix:

1. download the USB Driver via the Windows Android SDK

2.check your Android's VID and PID via Windows Device Manager

3. Open android_winusb.inf in the usb_driver folder and check that the VID/PID combo exists in the file.

4. If it doesn't, add it in into the respective INI section. For example, I added this for Droid Eris under the [Google.NTx86] section:

%CompositeAdbInterface% = USB_Install, USB\VID_0BB4&PID_0C98&MI_01

5. Save and reinstall the USB driver and that's it!

Proxoid users

If you're currently using Proxoid's windows drivers, they're not updated for Android 2.1. Try using the Android SDK version instead.

Also, you'll need to copy over adb.exe and AdbWinAPI.dll from the android sdk tools folder over to the proxoid-adb folder.

5 Comments »

Update

My patched version of Proxoid is available here: http://www.supermind.org/friends/SuperProxoid-debug.apk
Source code is available here: http://www.supermind.org/friends/superproxoid-snapshot-20101009.tar.gz

Note: I'm no longer using Proxoid, SuperProxoid, or even AziLink etc. I'm now using a rooted Android 2.2 which supports USB and wifi tethering! Way cool. Instructions here: http://www.supermind.org/blog/774/upgrade-your-htc-droid-eris-to-android-2-2

I've been using the excellent tether app, Proxoid to hook up my PC to my Android-based Droid Eris phone.

However, after a period of somewhat heavy usage/browsing, it inevitably barfs and refuses to take any more connections.

In Firefox, this appears as "Connection reset". In Chrome, pages simply refuse to load. The workaround I used for awhile was simply to stop the proxoid service on the Proxoid app, then start it again. This was fine (if somewhat annoying for surfing), but is a non-starter if you happen to be downloading anything at that point.

I decided to do abit of sleuthing, and installed the Log Collector app so I could see what was going on when Proxoid refused any more connections. Here's a snapshot of what the log file showed:

05-02 09:56:47.466 E/OSNetworkSystem( 6923): unclassified errno 24 (Too many open files)
05-02 09:56:47.466 E/OSNetworkSystem( 6923): unclassified errno 24 (Too many open files)
05-02 09:56:47.477 E/ProxoidService( 6923): null
05-02 09:56:47.477 E/ProxoidService( 6923): java.lang.NullPointerException
05-02 09:56:47.477 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.SelectorImpl.prepareChannels(SelectorImpl.java:223)
05-02 09:56:47.477 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.SelectorImpl.selectInternal(SelectorImpl.java:191)
05-02 09:56:47.477 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.SelectorImpl.select(SelectorImpl.java:167)
05-02 09:56:47.477 E/ProxoidService( 6923): at com.mba.proxylight.RequestProcessor$1.run(RequestProcessor.java:72)
05-02 09:56:47.477 E/ProxoidService( 6923): at java.lang.Thread.run(Thread.java:1058)
05-02 09:56:47.486 E/ProxoidService( 6923): null
05-02 09:56:47.486 E/ProxoidService( 6923): java.net.SocketException: Operation failed
05-02 09:56:47.486 E/ProxoidService( 6923): at org.apache.harmony.luni.platform.OSNetworkSystem.createSocketImpl(Native Method)
05-02 09:56:47.486 E/ProxoidService( 6923): at org.apache.harmony.luni.platform.OSNetworkSystem.createSocket(OSNetworkSystem.java:85)
05-02 09:56:47.486 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.SocketChannelImpl.(SocketChannelImpl.java:156)
05-02 09:56:47.486 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:79)
05-02 09:56:47.486 E/ProxoidService( 6923): at java.nio.channels.SocketChannel.open(SocketChannel.java:95)
05-02 09:56:47.486 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:131)
05-02 09:56:47.486 E/ProxoidService( 6923): at com.mba.proxylight.RequestProcessor.process(RequestProcessor.java:351)
05-02 09:56:47.486 E/ProxoidService( 6923): at com.mba.proxylight.ProxyLight$1.run(ProxyLight.java:127)
05-02 09:56:47.486 E/ProxoidService( 6923): at java.lang.Thread.run(Thread.java:1058)
05-02 09:56:47.496 E/ProxoidService( 6923): null
05-02 09:56:47.496 E/ProxoidService( 6923): java.lang.NullPointerException
05-02 09:56:47.496 E/ProxoidService( 6923): at org.apache.harmony.nio.internal.SelectorImpl.wakeup(SelectorImpl.java:332)
05-02 09:56:47.496 E/ProxoidService( 6923): at com.mba.proxylight.RequestProcessor.closeAll(RequestProcessor.java:323)
05-02 09:56:47.496 E/ProxoidService( 6923): at com.mba.proxylight.RequestProcessor.access$2(RequestProcessor.java:302)
05-02 09:56:47.496 E/ProxoidService( 6923): at com.mba.proxylight.RequestProcessor$1.run(RequestProcessor.java:246)
05-02 09:56:47.496 E/ProxoidService( 6923): at java.lang.Thread.run(Thread.java:1058)
05-02 09:56:47.496 E/AndroidRuntime( 6923): Crash logging skipped, already logging another crash
05-02 09:56:47.496 E/OSNetworkSystem( 6923): unclassified errno 24 (Too many open files)

Huh. Too many open files. That would explain the non-response of Proxoid.

Googling for the offending line (org.apache.harmony.nio.internal.SelectorImpl.prepareChannels(SelectorImpl.java:223), yielded this: Selector leaks file descriptors in Apache Harmony. Yikes! Bad news.

The reporter of the bug in Android's issue tracker said:

Thanks for the quick answer. The workaround I used is quite simple: create as few
selectors as possible and recycle them endlessly.

I took a look at the Proxoid source and realized the error was fatal. A new selector is opened for each new request.

Its technically not a bug in Proxoid, but its awful to have to keep restarting Proxoid.

So I decided to hack a replacement proxy server implementation for Proxoid which uses threads instead of non-blocking IO. Took me 2 days to work everything out, and I ended up using some code from Muffin Proxy Server but it looks like things work fine. I tested my app out on 10000 HTTP requests (Proxoid barfed after about 1000) and no errors thus far. 🙂

24 Comments »

Just discovered this gem here: http://www.betaversion.org/~stefano/linotype/news/110/

1. Android apps are developed in Java
2. Android itself is licensed in APL, but Sun's source is licensed in GPL.
3. Furthermore, Java is not open-sourced in mobile environments.
4. How did Google do it?

Turns out that Android

1. Uses Java as a development language but does not use Java bytecode or Java virtual machine for deployment.
2. Has a virtual machine called Dalvik which does not claim to be a JVM
3. First uses a regular java compiler to generate regular java bytecode (say, javac or the built-in Eclipse compiler) and then converts that bytecode into Dalvik’s bytecode (the “dx” tool does this: converts .class/.jar into .dex files)
4. Supports only a subset of Java SE (AWT/Swing and JavaME classes are omitted) classlib. Instead of using Sun's implementation, it uses Apache Harmony's implementation.

That's quite a tour de force.

No Comments »