In business, the absence of proof is proof.

This is how religion happens:

Customer relentlessly files tickets claiming subsystem is malfunctioning. Offers vague anecdotal evidence for failure state.

So, I go in and add copious logging; I am now able to see the entire state of the application. And of course, I test my debugging by poking random buttons; failure should occur (per customer) and in either case, I should see logged thingees.

You can guess what happens next: it works fine no matter how hard I give it the million-monkeys treatment.

I then sit and wait to see if any other random passers-by decide to use app. After a reasonable wait, I see no one has, so I declare it fixulated and close the ticket, noting the above (”cannot create failure state, marking fixed and closed.”)

OH NO NO NO, screams customer. We know it does not work. OK, I ask: what’s your proof?

Well, we have not gotten any notifications anyone has used it! We normally have lots! Ergo it doesn’t work.

I attempt to retort that all evidence shows 1)it does work and 2)you haven’t gotten any notifications from the application because no one’s using it presently. As soon as someone uses it (eg, me) it works. Quod erat demonstrandum.

That no one is using it is proof that it doesn’t work, I am told, at which point I feel like I am allowed to have a drinking problem and a bad attitude towards our users.

Our lovely data center.

Like, oh, everyone, we have all our hardware at a colo facility, managed by our bandwidth provider.

And like every colo facility, they have strict ingress/egress rules for persons and hardware. Want to rack up a server? Call the home office in Denver, request a ticket, go to Sterling with hardware, sign 20 different pieces of paper to get access to our racks. So fun.

It would make sense, if they had anything remotely resembling competence or caring. But they security are essentially untrained; they rarely speak english as a third or fourth language; no one cares enough to enforce policy, except when they do, IT WILL BE FOLLOWED TO THE LETTER. (strong correlation between english language fluency and desire to enforce regulations)

Perhaps a couple of examples will better illustrate how fucking stupid these asshats are:
1. My badge expired. All I had to do to get access was make a sad face, present a ticket ID, and ask nicely to be let in.
2. Then after my badge was 1)lost and 2)renewed, it was 3)lost again and I was not allowed, at all, to enter the hosting floor. That I had just used it once in the intervening space between 2 and 3, and that was less than 12 hours prior, was ignored.
3. Also, my picture badge was taken in 2004, when I started here. I was … larger. Recently they decided that I was not the person on the badge, because I am less large. “Look, thanks for the fantastic complement vis a vis my weight loss, but seriously pal, that’s me. OK?”

Note that in many of these instances there were serious, time-critical issues going on. No fuckin’ around, clock-is-ticking stuff.

I have to go back, again, today. I can’t wait to see what happens. It’s always an adventure.

Always Be Closing

So we’re hiring a salesman. (I don’t want to speak too soon, but in theory he’ll be officially on board today.)

This is a pretty big step for the company, and most interesting for me: at Barefoot we specifically resisted getting a salesperson.

The logic to this was, in part, “growing too fast will kill the company”. And no matter how much you want to argue it, you know it’s true: growth and stagnation/decline are embody mirror images of almost identical challenges (explosive growth versus not having enough money for infrastructure, for example).

In the BFSW case it wasn’t the horrible margins and infrastructure issues of hosting, it was more along these lines, we feared 1)having to bring on a dozen people of highly variable quality that we then had to manage, etc and 2)how long can we keep that up, keep everyone busy, etc etc.

We talked and talked and talked, and went on endlessly about growth vs stagnation, the challenges of growth, the very real risk that too much growth could burn things out, and so on. We finally decided to do it: it’s better to die winning the fight than running from it.

Of course, there’s so many variables at play here, I can’t even count them. If nothing else, the New Guy could suck as a salesman, or $PRODUCT will be far harder to sell than he thought, or he’ll find some other greener pasture, or he just won’t like it; and we’ll do another round of interviews and keep playing this game.

Still; it’s an exciting but scary challenge, one that I never got to face at BFSW or subsequent tech positions that shall go forever unnamed.

trolling DSL/cable IP blocks vs the rest of the internets

I checked my sshdfilter SSHD table the other day on 2 boxes: one at our data center, and another on our office DSL, each machine having gone online within a few hours of each other.

The latter had something like 4x as many entries as the former. Which leads me to my question: are the Bad Guys specifically trolling known DSL (and ostensibly cable) IP blocks, assuming more/easier pickings? The logs also show the usual massive run of web attacks, although I don’t have stats for comparison.

Is this normal?

Today, we sysadmin in Hades!

At semi-regular intervals, I take off my “web developer” hat, and put on my “sysadmin hat” for a day of totally awesome system administration.

This is the worst job in the world.

I basically spend my entire day - and usually a couple days afterwards - trying to figure out why the fuck I did what I did. Why the fuck would one box use an entirely different backup script, that, oh, doesn’t even work? When the fuck did someone change database dump script? Hey, why is that box so behind on updates? And so on, and so on.

These days are like penance for all my sins: fiddling with shell scripts that are by all rights a 5-minute quick hack, but became “production”.

And yes, I’m going to hell for using a ‘300′ cliche. I don’t care any more.

Payment Card Industry Security Standards are a fucking joke

46 Million credit card numbers stolen:

NEW YORK (CNNMoney.com) — The retailer that owns the T.J. Maxx and Marshall’s clothing chains said nearly 46 million customer card numbers were stolen from its computers over an 18-month period and said the total number of stolen cards may never be known.

A couple years ago, a number of our merchants - who have not in their decade of business made even 4.6 million total sales - were required to implement lots and lots of new business practices to adhere to the Payment Card Industry Security Standards, else they lose their merchant accounts.

Obviously this whole industry-wide push has yielded incredible results. I said it at the time, and I’ll say it again:

• PCI compliance is a shakedown
• Only small merchants were “targeted”, no large retailer will ever suffer the wrath of Visa/Mastercard/Amex
•  Nothing in PCI will do much to stop security problems, anyway

The entire system is a giant CYA and has nothing at all to do with genuine security. But great PR is better than real work.

Overhead in the office

(in discussing what to do after a customer has done something very stupid, repeatedly)

TheBoss: I think it’s time to drop the hammer on person.

Me: Awww, how come I never get to do that?

TheBoss: (pauses) Because you’d actually go find them and take a hammer to them.

Startupping: Lessons Learned

From http://www.startupping.com/2007/02/20/best-and-worst-decisions-part-1/

I’ve made loads of mistakes so I’ll try to think of one with a good lesson for startups - one of the biggest mistakes I made in a previous company was accepting a high dollar contract once for something that wasn’t core to the vision of the business we were running at the time. While the revenue initially feels great, there’s nothing worse than pursuing a piece of business that isn’t core to the startup’s vision. Lesson learned - once you decide what it is you are going to do, don’t pursue efforts that distract from the vision.

This is  pretty much what we do every few months. We’ve been trying to launch a new product since October. Every time the same damn thing happens:

1. Set aggressive schedule
2. Realize schedule is too aggressive/we’ve underestimated requirements
3. Pad out schedule some
4. CRISIS! Schedule is blown
5. Meander off on tangents
6. Reconvene to try to get back on track
7. GOTO 1

We’ve tried 3 launches of $NEWPRODUCT: one in late summer of last year, then October of last year, then this month.

Guess what we haven’t launched.

The biggest problem is we accept work outside our core business: because it’s a lot of money, typically, but also because small businesses (as opposed to “startups”) tend to have a deep-set fear of turning down work. We have to start doing that, or we’ll never launch, and if we never launch we are possibly depriving ourselves of even greater growth (revenue and overall).