Skip to main content

Songs on the Security of Networks
a blog by Michał "rysiek" Woźniak

Startup Weekend Network Fun Fun Fun

This is an ancient post, published more than 4 years ago.

As such, it might not anymore reflect the views of the author or the state of the world. It is provided as historical record.

To all the newcomers here – this is alpha version of my brag, and things will break. Please use Firefox or Chrome/Chromium to have the best experience; Opera will work too, but will not be as nice. Rekonq, Konqueror, other KHTML/Webkit-based browsers – newer version probably needed. To all the Internet Explorer and Safari users out there – please get a real browser, it’s free and it’s fun.

During the weekend I was responsible for the network and tech support at Startup Weekend Warsaw co-organized by my lab at Warsaw University of Technology. I cannot say I did a stellar job, but then again, I cannot say I didn’t try and do my best to keep the network alive and kicking. So…

The Scenery

100+ participants. Every single one of them with at least one laptop. Most - with some mobile, WiFi-enabled device in their pockets, too. Many with some third device - tablet, second mobile, etc. All in all, probably more than 200 devices trying to connect to the Internet. Quite a bunch.

The Setup

2 Access Points (Linksys WRT-54GL; one with vanilla Linksys software, the other with DD-RT), two 8-port gigabit switches, loads of cable, and the Faculty infrastructure. And it’s always about the infrastructure, ain’t it.

The Plan

The Plan was simple enough and looked quite well:

  • set-up two different WiFi networks, on different, separated channels;
  • get as many people on the wired network as possible.
  • ???
  • PROFIT!!

As always, the “???” turned out to be the crux of the whole thing.

The Unexpected

As per Faculty Network Policy, many outgoing ports were blocked. Obviously, in that NAT-ed network, all the incoming ports were filtered. I decided to set-up a tunnel (an SSH-based VPN) for those few of our users that would need some “exotic” ports (like, oh you know, 25/tcp if they would fancy sending an e-mail). That was supposed to be far from mission-critical and just a courtesy towards the technically-inclined guys and gals in the room – so, basically, 90% of them.

However, it became uber-critical as soon as it turned out the (important) live audio/video stream that was supposed to allow more people to participate on-line actually uses some of the blocked ports. Whoopsie! The quick-and-dirty solution became a very important piece of duct-tape.

Fun with Streams

And there were loads people watching this stream too! Problem was, many of them were in the very room the stream was transmitted. Now, sending a video stream was enough of a network hog to cause minor hiccups; when people started watching it within the same network, basically all hell broke loose…

Oh, and let’s not forget the great job Skype was doing to help our network tank even deeper. Yay for that.

The Fa(c)ulty Infrastructure

To be honest we had some real faith in the Faculty’s solid backbone. And with good reasons too. It is a solid backbone, so why shouldn’t we? Ah, faith, you are a funny thing. There comes a moment that reality catches up and, say, the Faculty’s DHCP server goes down. Good for us we had a nice Ubuntu box (yes, the one with the SSH tunnel/VPN running). 5mins with apt-get and dnsmasq.conf and we were back on-track to the next failure in the string.

WiFi Mavericks

Well, obviously, the wireless quickly started getting quirky. As in, not working properly. Or at all for that matter. When suddenly 100 devices try to connect to a single AP in a matter of minutes, the AP will go down in a matter of those same minutes. Vicious circle.

So people started using 3G connections, which would not be that bad, as it would lessen the traffic on the poor battered APs, right? If only those were used via Bluetooth or USB. But guess what? Setting up your own ad-hoc WiFi mininetwork is sooo coool, right? Hence, suddenly, we had about 15 different ad-hoc networks interfering with the two Startup Weekend official WiFi nets. Guess what, that was not helping.

How it all played out?

or “putting the fires out”

To be honest - not well. There were simply too many points of failure. Too many fires. Often times the APs got in some strange mode in which connectios already established work passably, but no new devices were able to connect. Should we reboot such an AP to get the new devices on-line, or just go with the flow and let the already connected use the network without interruptions? Damned if you don’t, damned if you do.

Lessons Learned

So, “mistakes have been made”, moving on with the knowledge. In particular:

  • QoS! next time each and every single user will get a dedicated, albeit small, bandwidth channel.
  • less security can buy quite a nice amount of reliability; seriously, we did not need WPA here, we could have gone with WEP – or no security at all.
  • in-house as much as you can: get your own network segment, your own DHCP server, etc. – this way at least you are in control if something goes awry; and believe me, it will.

We got a few things right, a bit more on that further on down.

Blaming and Name-calling

We already know I was responsible for WiFi, but far from being able to do it all by myself. Special thanks for all the hard work go to Piorek and Karolina. Piorek was helping me all the time with tech stuff (and doing a great job); Karol was the bureau and chancellary, making everything go as smoothly as possible.

Many thanks to Kamila, Konrad and Krzysiek for the great atmosphere at the conference. You guys should have been pissed a few times, but weren’t, and that went very far in helping us deal with the Wireless Notworking rather than interpersonal stuff. Seriously, to all the conference organizers out there – take heed, as this may well save your WiFi!

Last but not least - [@zstanska](http://twitter.com/zstanska) and [@mpaluchowski](http://twitter.com/mpaluchowski), doing the social media and video streaming, were usually the first to nag inform us about any problems. And in style! Also made fun of us on twitter. Really, you guys could have used a better service.

Finally, heartfelt thanks to all the people at and around Startup Warsaw.

Icing on the cake

Microsoft reps doing a presentation on a Linux box (that was our presentation box, simple as that)… I am not even sure they knew that, to be honest. Ah well, fun anyway. If and when I get the video, I will drop it here.

Actually, quite a few people had Linux on their lappys. Interesting times.

Follow Up

Today Shot sent a great article about making conference WiFi work. Better late then never – but hey, turned out we actually did many, many things right! I’ll try writing a follow-up bragpost on that later, when I sleep a bit.