This is an ancient post, published more than 4 years ago.
As such, it might not anymore reflect the views of the author or the state of the world. It is provided as historical record.
To all the newcomers here – this is alpha version of my brag, and
things will break. Please use Firefox or Chrome/Chromium
to have the best experience; Opera will
work too, but will not be as nice. Rekonq, Konqueror, other KHTML/Webkit-based browsers –
newer version probably needed. To all the Internet Explorer and Safari
users out there – please get a real browser, it’s free and it’s
fun.
During the weekend I was
responsible for the network and tech support at Startup Weekend Warsaw
co-organized by my lab at Warsaw
University of Technology. I cannot say I did a stellar job, but then
again, I cannot say I didn’t try and do my best to keep the network
alive and kicking. So…
The Scenery
100+ participants. Every single one of them with at least one laptop.
Most - with some mobile, WiFi-enabled device in their pockets, too. Many
with some third device - tablet, second mobile, etc. All in all,
probably more than 200 devices trying to connect to the Internet. Quite
a bunch.
The Setup
2 Access Points (Linksys WRT-54GL; one with vanilla Linksys software,
the other with DD-RT), two 8-port
gigabit switches, loads of cable, and the Faculty infrastructure. And
it’s always about the infrastructure, ain’t it.
The Plan
The Plan was simple enough and looked quite well:
- set-up two different WiFi networks, on different, separated
channels;
- get as many people on the wired network as possible.
- ???
- PROFIT!!
As always, the “???” turned out to be the crux of the whole
thing.
The Unexpected
As per Faculty Network Policy, many outgoing ports were blocked.
Obviously, in that NAT-ed network, all the incoming ports were filtered.
I decided to set-up a tunnel (an SSH-based VPN) for
those few of our users that would need some “exotic” ports (like, oh you
know, 25/tcp if they would fancy sending an e-mail). That was supposed
to be far from mission-critical and just a courtesy towards the
technically-inclined guys and gals in the room – so, basically, 90% of
them.
However, it became uber-critical as soon as it turned out the
(important) live audio/video stream that was supposed
to allow more people to participate on-line actually uses some of the
blocked ports. Whoopsie! The quick-and-dirty solution became a very
important piece of duct-tape.
Fun with Streams
And there were loads people watching this stream too! Problem was,
many of them were in the very room the stream was transmitted. Now,
sending a video stream was enough of a network hog to cause minor
hiccups; when people started watching it within the same network,
basically all hell broke loose…
Oh, and let’s not forget the great job Skype was doing to help our
network tank even deeper. Yay for that.
The Fa(c)ulty Infrastructure
To be honest we had some real faith in the Faculty’s solid backbone.
And with good reasons too. It is a solid backbone, so why shouldn’t we?
Ah, faith, you are a funny thing. There comes a moment that reality
catches up and, say, the Faculty’s DHCP server goes down. Good for us we
had a nice Ubuntu box (yes, the one
with the SSH tunnel/VPN running). 5mins with apt-get
and
dnsmasq.conf
and we were back on-track to the next failure
in the string.
WiFi Mavericks
Well, obviously, the wireless quickly started getting quirky. As in,
not working properly. Or at all for that matter. When suddenly 100
devices try to connect to a single AP in a matter of minutes, the AP
will go down in a matter of those same minutes. Vicious circle.
So people started using 3G connections, which would not be that bad,
as it would lessen the traffic on the poor battered APs, right? If only
those were used via Bluetooth or USB. But guess what? Setting up your
own ad-hoc WiFi mininetwork is sooo coool, right? Hence, suddenly, we
had about 15 different ad-hoc networks interfering with the two Startup
Weekend official WiFi nets. Guess what, that was not
helping.
How it all played out?
or “putting the fires out”
To be honest - not well. There were simply too many points of
failure. Too many fires. Often times the APs got in some strange mode in
which connectios already established work passably, but no new devices
were able to connect. Should we reboot such an AP to get the new devices
on-line, or just go with the flow and let the already connected use the
network without interruptions? Damned if you don’t, damned if you
do.
Lessons Learned
So, “mistakes have been made”, moving on with the knowledge. In
particular:
- QoS! next time each and every single user will get a dedicated,
albeit small, bandwidth channel.
- less security can buy quite a nice amount of reliability; seriously,
we did not need WPA here, we could have gone with WEP – or no security
at all.
- in-house as much as you can: get your own network segment, your own
DHCP server, etc. – this way at least you are in control if something
goes awry; and believe me, it will.
We got a few things right, a bit more on that further on down.
Blaming and Name-calling
We already know I was responsible for WiFi, but far from being able
to do it all by myself. Special thanks for all the hard work go to Piorek and Karolina. Piorek was helping
me all the time with tech stuff (and doing a great job); Karol was the
bureau and chancellary, making everything go as smoothly as
possible.
Many thanks to Kamila, Konrad and Krzysiek for the great atmosphere at
the conference. You guys should have been pissed a few times, but
weren’t, and that went very far in helping us deal with the Wireless
Notworking rather than interpersonal stuff. Seriously, to all the
conference organizers out there – take heed, as this may well save your
WiFi!
Last but not least - @zstanska and @mpaluchowski, doing the social
media and video streaming, were usually the first to nag
inform us about any problems. And in style! Also made fun of us on
twitter. Really, you guys could have used a
better service.
Finally, heartfelt thanks to all the people at and around Startup
Warsaw.
Icing on the cake
Microsoft reps doing a presentation on a Linux box (that was our presentation
box, simple as that)… I am not even sure they knew that, to be honest.
Ah well, fun anyway. If and when I get the video, I will drop it
here.
Actually, quite a few people had Linux on their lappys. Interesting
times.
Follow Up
Today Shot sent a great
article about making conference WiFi work. Better late then never –
but hey, turned out we actually did many, many things
right! I’ll try writing a follow-up bragpost
on that later, when I sleep a bit.