Yeah, I’ve been slacking pretty hard on this web-log front. Lots of reasons, none of them very good. I’m hoping to get back on the horse a little bit now — I had, in fact, hoped to do so yesterday — so we’ll see if I have any readers left!
Some stuff I’d like to write about over the next week or so:
Or, you know, I could just disappear again, under the weight of my own mounting ambition and inability to manage my time. Were I you, I’d place bets very carefully, if at all.
I’m going to be speaking about web futures at the Desktop Developers Conference on Tuesday, if people find themselves in Ottawa and want to waste an hour of their lives listening to me. (Actually, I think it’ll be a pretty good talk; Brendan and I are both quite keen to get into this discussion with the rest of the open-desktop community, and there’s a lot of interesting stuff to discuss.)
After that, on August 6th, I’ll be speaking at the Mozilla Developer Day about Mozilla 2.0: what we need to do, what we know we’re not going to do, what we’re still trying to figure out, and how we’re going about making these decisions. If you’re in the Bay Area, it’ll be well worth your $25 to come and talk with us about this, or listen to the other talks. I hear the Googleplex is pretty sweet, too.
I had a hard time getting the network from the Ximian/Novell offices to play nicely with the Oracle VPN, but I did manage to fix two bugs that were threatening to slip the Mozilla 1.7 release. It’s good to be back!
I’m in Montreal today, talking about calendar protocols with some Oracle co-workers and Lisa from OSAF. I don’t know a lot about calendaring, but I know a little bit about what makes adoption easier or harder for open source clients, and I’m learning a fair bit here.
After a nice long meeting, which in turn followed a 7:30 AM flight, I was starting to flag a bit, so Lisa and I took a nice hike on the mountain and talked about the joys of open-source project management.
Now, to dinner.
I finally got my offer package today, and Tyla helpfully Purolatored it off to the processing centre in California — Jeremy’s advice was “send it exactly as it says; don’t fight with The Process” — so I’m a small amount of paperwork from being an official Oracle employee. I’m pretty psyched.
Beltzner and I celebrated with champagne and wings, after a start-of-season Ultimate clinic, and before joining some other friends in the viewing of a pretty-mediocre Angel episode.
Then I bit my tongue harder than I’ve ever bit it before, and now have a bruise about the size of a dime. Or, perhaps, the size of a molar.
< dria> we should so totally rule the world
< phik> if we'd just stop playing russian roulette with each others' careers
This Friday, as some of you are aware, is my last day with CFS. I'm not going into the whys and wherefores of the decision right now, if ever, but your guesses are probably not all that far off the mark. It was a very fun ride, and I learned a tremendous amount, which is basically the sum of my success conditions for a job; I wish the CFS crew nothing but the best, of course.
I am presently entangled in a mutual courtship with a large software vendor who is looking for a strong open source strategist and technologist to lead their nascent efforts in the whole "open desktop space". None were available, apparently, so they're talking to me.
The aforementioned dissonance has been resolved, in a number of ways, and I thank everyone for their support and concern. I'm not going to write much more now, because I think I'm going to be writing a lot about work things in the next few weeks/months, and I'm on vacation now.
One of the strangest vacations of my life, in fact, but that's a story for another day.
Ever get totally kicked in the teeth by a comment (or short email) from someone which reveals an enormous gap between how you percieve something and how they percieve something? Ever have that happen a few dozen hours after the last conversation with that person, in which it very much seemed that everything was well-aligned, and things were proceeding well?
Yeah, me too.
(Honestly, this is the biggest "what the hell?" moment I've had in quite some time, and it's not like my professional life has been devoid of this sort of disconnect over the last year or two. I'm flabbergasted.)
Madhava came over yesterday and I showed him the interface for the web tool I’m working on. He didn’t throw up even a little, at least while he was still here. We had quite a productive chat about the data I need to present and what my constraints are — number one with a bullet: web tool, usable in at least Mozilla + IE, probably Safari as well — and he brought me an outrageously cool book chock full of blurbs on different, well, information graphics. Soon I’ll be able to talk like a real interface devotee, and not just someone scrambling against the clock to produce anything usable.
I’ve been working on Lustre recovery for quite some time. Probably a year and a half, I guess, and there was some good work done in there. Not all of the work was good, mind you, but I’m not totally ashamed of what I accomplished. After a few false-starts at the process, though, responsibility for that subsystem — some people would consider it an aspect because it cuts across so many otherwise well-layered subsystems, I suppose — has passed on to one of my co-workers. He’d assisted me a fair bit in the past, and I think he’ll do a fine job. Which means I won’t have to, at least as far as recovery is concerned, so I’m pretty psyched.
Instead, I have taken over responsibility for a few other Lustrey things. Don’t worry, I”m not hurting for things to do.
The first piece — I think this is in chronological order by date of assignment, but if it’s not…\/\/hatever — is the design and development of at least two tools to assist in the configuration, management and monitoring of Lustre installations (one text and one graphical). Right now it’s more of a manual (and therefore error-prone) process than it really should be, and it requires that the administrator know far more about our configuration internals than is ultimately desireable. Some of our larger clients have written their own layers of shell scripts and whatnot atop our tools and their own monitoring software, but we’re looking for something a little more general and a little more polished. (With no offense intended to the authors of those tools!)
So far, I’ve got the framework for the text-based utility limping along, loading configuration information, and poking tentatively at the few nodes in my dogfood cluster.
In addition to that, I’m also leading the charge on some new training programmes, including material for students and trainers, actually delivering that curriculum (at least in the near term), and helping schedule/plan/fill on-site and “open” training sessions. Beth and I are in the middle of a massive rewrite of our presentation materials this week, and I hope to have that done and polished nicely in time for my first class (tentatively scheduled for Feb 2-3). I’ll be delivering this course at least a few times, and possibly many more, before we hand it off to a full-time training-delivery type, so the materials had better be good enough to stand in front of for 16 hours without making me want to cry. Likely to be a lot of travel involved, it looks like.
Of course, I’m also still managing parts of our QA work, so I have to scurry around every day and think of new ways for Coop to make me look good.
Looks like we’re on track to make Apple customers kinda sad when they type ls. (Actually, there’s some excellent work underway to improve that dramatically for the next release after Lustre 1.0.x, which I would like to think was in some way inspired by my previous work in this area. I don’t actually think I contributed a lot to the rocking LVB work that Peter and Phil and Zach are doing, but I would like to.)
I finally found the other bug today, and more or less fixed it. Nobody in particular will care, I’m sure. (I say “more or less” because I’m not sure if people are going to go for my solution. I’m ready to wash my hands of it, though.)
So, we’ve been having a little bit of trouble getting our test suite running and passing on a new cluster of late. I should say that Robert has been having trouble, because I don’t exactly have access to the cluster right now, due to a Marx Brothers-grade set of difficulties with my cryptocard and whatnot. We also have some new things in the mix this time, like a mostly-righteous new configuration system, a totally alien cluster organization, all the usual fun you want right before an acceptance milestone.
Robert has been describing his troubles to me over the last little while, since I’m the other guy who has an oversized recoverythalmus. I haven’t really been poring over the code like I would like to, in this situation, but I have been thinking about it in the back of my head while I go about my other business. Last night, we got a clue in the form of some atypically-cryptic console spew:
[lots of this sort of noise]
fa3cac65 <[mds]mds_destroy_export+1e9/394>
fa1c6adc <[obdclass]g_uuid_lock+18c/24c4f>
fa1c69a0 <[obdclass]g_uuid_lock+50/24c4f>
fa1a9858 <[obdclass]obd_destroy_export+118/186>
fa1bfeb5 <[obdclass]__kstrtab_obdo_to_ioobj+4bbb/b006>
fa1a691b <[obdclass]__class_export_put+7f/250>
fa1bfe15 <[obdclass]__kstrtab_obdo_to_ioobj+4b1b/b006>
fa1c016d <[obdclass]__kstrtab_obdo_to_ioobj+4e73/b006>
fa1a7dd7 <[obdclass]class_disconnect+1b3/5e4>
[and so forth]
From that, the answer is obvious, of course: connect and disconnect are racing, and the bitmap maintenance in mds_client_free isn’t safe against such races. Robert’s whipping up a patch now, and I am going to treat myself to a tasty apple.
5 computers arrived for me today (3 of them mine, for dogfood work; one for each of Andrei and Vlad), and I set them up. Holy crap, is it loud and hot in this office. My machines seem to be perfectly happy — kudos, indeed, to the people responsible for the VNC installer in Fedora Core and for open-carpet! Andrei’s machine isn’t so happy, so I think he’ll be getting someone to replace part or all of it.
Tomorrow I get to finish setting up Lustre and pdsh and distcc and all the other nice toys for making use of a handful of machines, and then beat the crud out of them. And maybe find a way to make them quieter, good heavens.
Update: pdsh and distcc are done, and it’s a beautiful thing.
: SPECS; pdsh -l root -w alpo,purina,iams uptime iams: 10:40:41 up 2:20, 1 user, load average: 0.00, 0.00, 0.00 alpo: 10:39:48 up 5:53, 1 user, load average: 0.06, 0.16, 0.08 purina: 10:39:29 up 3:15, 1 user, load average: 0.10, 0.22, 0.09
The demo is shaping up nicely, and some of the far-flung CFS crew are chipping in to extend the geographical reach of the filesystem in question. I put together a little ball of networking twine and UML bubblegum to contribute a Canadian mount:
bash-2.05b# df -h /mnt/lustre Filesystem Size Used Avail Use% Mounted on sc03_dual_eth3 15T 1.3T 12T 10% /mnt/lustre
Robert and Eric managed to also join us from California and Bristol, but the main demo component will be joining the party over the impressive SC2003 WAN.
I fixed some real bugs today, too, so a good show all around.
A handful of my esteemed co-workers are down at Supercomputing 2003 working on, among a tutorial and other things, our contribution to the crazy Bandwidth Challenge shenanigans.
Apparently the tutorial — in which a small horde of people is being tutorialized as I write this very entry — is going quite well, so things are off to a good start. It’s pretty exciting, even for those of us who are cheering from megametres away.
Now that people are doing real sciencey stuff on top of our baby, I think things are going to pick right up around here. Not that there’s really been a lot of thumb-twiddling recently anyway, but.
Addendum: zoom.
Bah:
ASSERTION(rec->ur_fid2->id == inode->i_ino) failed
Not a hard problem to fix — I can already think of not one but two solutions to the problem, and I’ve only been awake for an hour! — but the timing of this find truly, truly sucks.
Furthermore: bah.
I owe myself more and better entries, and I owe them with usurious interest. They’ll come soon, because I’m coming out of a dark, dank, productive place, in which I’ve spent most of the last month.
This won’t mean anything to any of you, but please rest assured that it means the world to me:
status: COMPLETE recovered_clients: 714 last_transno: 10081296 replayed_requests: 61
I lost an entire day today. The pathology is all very geeky and work-specific, but the effects are simple enough: I want to punch the universe until it stops twitching, and then set it on fire.
Thank you for your attention to this matter.
Some parts of my day were lost to iterating over a crappy test suite — a crappy test suite that I wrote back when it was last really critical that this stuff get fixed up and tested, so I can’t really get too indignant about that part — and some parts were lost to some suboptimal logging and power-management infrastructure on the cluster in question. But a lot of it — the bulk, certainly — was lost to the fact that there was a change in the software I work on. Before the change, you could, in this configuration wooble, use pretty much any name you wanted, as long as you used it consistently throughout the aforementioned configuration wooble. After this change, there is only one legal name.
Picking the name incorrectly does not result in an error message at name-selection time. Nor does guessing the wrong name give you a failure to start up. No, a name mismatch just gets you a nice crash after you start to talk to the filesystem — a crash which is a little tricky to diagnose when you have logging and crash-dump tools that are a little less than perfectly useful.
At least nobody will get beaten by that in the future.