Category Archives: IT howto

Proxies must not retry

You live in the cloud. Your app lives in the cloud. Mostly. You’ve decided to add access controls via a simple proxy. Your service is supposed to have “100%” uptime, so of course the proxy has to have “100%” uptime.

So far so good – except that the back end only has 99.9% uptime and your stupid ops people have set up alarms that check service uptime via your proxy. Since you don’t want to get dinged you figure you’ll retry. No alarms, no problem. Right?

Truth is you’ve just made your app slower. Probably a lot slower. And more expensive. And less stable.


Look at the data

Have a look at this picture. This is a test for a proxy that retries after 15s.


Let’s focus on the orange data. You’ll have to trust me when I say there is orange dots under the green dots. What you see is that the retry works really well: typical response time is about 2s and if that fails we get responses after about 17s (15+2) and if that fails we get responses after about 32s (2*15+2) and if that fails we get responses after about 47s (3*15+2). This is great! The proxy works!

Does it though? What should the client do? Should it wait for 50s? Or should it retry retry 25 times after 2s in the hopes that a single call will take the expected 2s? ? 10 times after 5s to account for some spread? Exponential backoff?

Based on the orange lines the client should absolutely retry every 3-5s. Of course that will kill your proxy and back end because each of the “timed out” calls will still go through the full proxy/back-end retry cycle. You just DoSed yourself.

Or course the blue data is more realistic. Under load there is actual spread. Some calls really do take up to 15s. So really you want exponential backoff. But even now you are abandoning calls to the retry pattern and DoSing your self. Not as badly but still.

In both of the above cases you client contains retry code. Now, why would you have retry code in your proxy?

I don’t believe you!

Ok. Just for you I have created this cool little toy on GitHub which allows you to walk through this step by step. Let’s say your server takes at least 2s to respond and at most 6s. Let’s model this as a gaussian because they are pretty:

Bad retry example
The blue line shows instantaneous probability that your request will be served at this time. The green line is the integrated probability, meaning that your request will be served by this time. Basically at 6s it is all but guaranteed that you received a reply.

So far so good. Now let’s have a look at the red line and what happens if we retry. If we retry early then we give up on any chance of the old request being fulfilled and start the wait again at the beginning. What this shows very nicely is that for any retry before you are guaranteed completion at 6s your performance will get worse.

How’s that different from the client doing the retry? Admittedly it isn’t. Except the client now has to wait until it’s guaranteed that the proxy would return!


Microcorruption uctf

A fried alerted me to one of the hackaday CTF games: I think every programmer, nay anyone using a computer, should play this!

Well, ok so this spoke to my obsessive nature and I really wish something this cool had existed when I was young, poor, and had the spare time to engage in some real hacking. Not that any of us did. Ever.

I admit it speaks to obsessive personalities and you probably won’t make it past the first 5 or so levels unless you have that obsessive streak, but even if you don’t: it’s a GAME. It has LEVELS. It TRAINS you and levels get progressively harder. And it has this cool hall of fame that shows you how much better you did than everyone else. And if you finish the tutorial and the first level you are already in the top 50% :)

And there’s lots of people playing it so you *can* get help if you ask. In fact it’s so popular that people are creating mods / tools / plugins to play this game:

Keeping track of cool stuff

Today I had time in front of a real computer and so I went down the rabbit hole of . There’s amazing stuff here and like always when I see amazing stuff, I would like to keep track of it!

Like any other site that want to sell you stuff, they have a wishlist and favorites … and that’s actually very well implemented … but it still requires for me to be logged into their site and doesn’t really lend itself to sharing …

This seemed like an obvious case of Pinterest to the rescue …

Now getting that to EMBED … that’s another story entirely! Once again being at the bleeding edge of WordPress is working against me because the recipes people have shared don’t work in 3.6.1 (yet).

Probably the cutest solution to the problem was using the Flickr plugin with the Pinterest RSS feed, so if you have a Flickr plugin that works … great.

The second best suggestion I found was this: use the Pinterest widget builder and embed that into your site. This is great but I want this to work whether I am displaying just one post or many and it’s quite specific about not loading the widget multiple times. What to do …

… I ultimately decided to embed this is the theme … but the suggested “easy” execution scheme is mindboggling. It suggests to put the script tag just above the /body tag, apparently because it needs to execute onLoad. Well, it does seem to work in the footer (Appearance – Editor – Footer) … but the page rendering is … less than desirable.

Embedding the “Advanced” version in the HEAD section work much better …

Is it safe? I don’t know but I don’t keep anything high value here, right?

Mont Blanc Corvids

After all that pain of embedding images / galleries it seems anticlimactically simple to embed videos into a blog …

  • set up a youtube account … it’s as easy as logging in – google will know who you are, I am sure
  • upload video … it’s as easy as drag and drop
  • drop the youtube url directly into WordPress …

And …

But the best part of it is how smart youtube is about the video … the video above was a freehand 3gp from an ancient android phone and you can see that it’s portrait, not landscape. Youtube notices this! It also offers post processing such as de-shaking – ok, I could do better with virtual dub but not at the cost of clicking a button!

Youtube plus WordPress … so easy!

Looking for galleries (2)

Spent a little more time today trying to figure out how to make the gallery thing work. It’s weird, hoarding pictures seems like one of the most common things people do: snap a million pictures, scribble some notes on them to remember where they were taken, laugh about the most compromising ones with friends and make a couple of prints of some special ones for the dresser …

… yet there seems to be practically nothing decent out there to do this with.

Here is what I figure the deliverables for common photo hoarding should be:

  1. Easy to get pictures out of camera
  2. Easy to bulk caption pictures
  3. Easy to do the basic cleaning operations (rotate, crop, red-eye, white balance)
  4. Easy to organize pictures
  5. Easy to share pictures with everyone, ideally on the couch, yours or theirs
  6. Easy to embed them into other projects
  7. Easy to back up.
  8. Not subject to changes in web software version or at least in a data format that’s easy to bulk re-ingest somewhere else.

Photo-albums  - you know, paper and stuff – can do 2,4, 5 and 8. Slides can’t actually do any of the above.

Digital age to the rescue … you’d think …

Facebook, the biggest photo sharing site out there, can do 1,2 and 5. Picasa can do 1-3, 5, and maybe 6 and 7. I haven’t looked at flickr but I suspect it’s going to be similar.

As for offerings that a normal person can install on an amazon or digital ocean instance … gallery/gallery 2 will do 1, 2, 4, and 5 but is a nightmare of bugs. WordPress can do 1 and 5, but unless you are Steve Jobs and don’t believe anyone needs folders it’s probably no good for more than about 5 pictures, not to mention that apparently the code changes so much between versions that it’s a FEATURE of the plugin system to collect user reports on what plugin works with what version.

Why is this so hard?!? And I am not talking about building lightboxes, which is hard but basic fuctionality.

So where am I at now?

Sigh …

Looking for galleries

What is the point of having a blog? Ramblings. What makes people like ramblings? Pictures. So how to get pictures into WordPress.


Apparently you can just use the native gallery and it’s actually fairly easy and functional:

  • in the editor click “add media”
  • upload media
  • click “create gallery” and select media you want in the gallery
  • insert gallery

Cool but boring.


The jetpack plugin promises lots of goodies. Maybe I don’t get it but the gallery seems to simply not work at all. The mobile theme is a nice addition but since you can’t customize it to match the main theme also falls short …

… to be continued

(Re)building web presence: wordpress

The Cloud Formation Way

For the last couple of months I have wanted to try WordPress, and specifically the Amazon Cloud Formation template. I initially tried this from my iPad while on vacation but there are some … issues. It could probably be done but my would you be in for pain!

The process is slick and fast although you will need to already have a key pair installed and probably should know how to log into your instance afterwards. Note that the default instance size is m1.small which is likely overkill for your test site.


After confirming that wordpress looked like the right tool for the job I really don’t want to have yet another instance … the monthly fees do add up … so how about installing on my existing instance. This looks easy:

  • make sure you have mysql php and a web server
  • download the tarball to web root
  • follow the install instructions

It gets a little tricky if you happen to be using nginx … in that case you probably want to also consult these:


I would post a link to the result … but you are looking at it ;)


In the process stumbled on which seems to be trivially cheap in comparison to EC2 … likely a future post