Database upgrades suck when you have to maintain them for multiple DBMSes. In working on the new installer (and associated updater refactoring), we’ve come up against a very annoying wall: maintaining versioned updates for databases gets to be difficult. With varying levels of support, the only schema that is guaranteed to work–provided someone didn’t break trunk–is mySQL. We maintain a list of of patch files to be applied, in order, on each upgrade. This list is kept up to date for mySQL; SQLite generally just works with mySQL’s syntax so it’s pretty close behind in support. Postgres tends to lag a few hours or days (and maintains its own set of update sequences, actually). Oracle doesn’t do patch files, but does update its default schema…so updates aren’t really possible except manually. Ibm_Db2 is all but abandoned, and Mssql was removed.

I briefly experimented with abstracting the whole schema to a big array of table definitions, but this was going to take too much work so the work was split off into a branch. One day I’d like to go back to this–done properly, it would cure our schema woes forever. Left with no other solution, Max forced the current updaters to run from the web, grabbing the output and throwing it back at the user. This is bad for a couple of reasons.

1. Updates can take a long time to run, especially if you have big tables. The fact that we run every update every time you try to upgrade wastes cycles.
2. It makes it impossible to localize along with the rest of the installer, all of the updaters output is hardcoded.
3. Without a clearly defined interface, it’s hard to get information back to the user and they’re left reading the output from update.php. Did the script warn or fail completely? Can we proceed?

So I decided to refactor things a bit. First I changed the updatelog table to contain an optional ul_value column (it already had ul_key). I moved the array of updates into their own class, so we can at least visually separate the list from the implementation. I then moved some of the core logic out of updaters.inc into a new Update class. Issue number one is pretty much mitigated now, since the new class logs which updates are performed, so it won’t needlessly repeat updates. Issues 2 and 3 are more possible now, since we could (haven’t yet) implement an output callback system like I did for Installer::performInstallation().

SQLite was easy, and Oracle just has a dummy implementation for now. Postgres is another beast entirely. Like I mentioned above, it maintains its own update sequence. Parts of it make sense, parts of it don’t, but none of it is designed like the others. Basically needs to be rewritten entirely.

For the new-installer to be complete by the time 1.17 rolls out, we need to pick up the pace. The time for new features is pretty much over and we need to start closing up the last few bugs. Updates are a blocker still. They’re a bit better than they were, but it’s still got a lonnnggg way to go before we’re done maintaining patch files for each DBMS. As always, bugs in the new installer can go here or in Bugzilla.

Sometime in 2004 or 2005, I first head of Wikipedia. It was this online encyclopedia with a lot of interesting articles. And the links! Oh the links; they transport you quickly to topics you’ve never even heard about. I soon learned that this marvelous resource was editable by people just like you an I. I was already familiar with free software, so the crowd-sourcing aspect was a concept I was already familiar with.

Becoming more involved, I joined the ranks of the editors and then administrators seeking to organize, revert and discuss the ever-growing content that formed the encyclopedia. My use of e-mail skyrocketed, as I found myself participating in long-winded mailing-list discussions about the intricacies of fair-use media and whether or not joke articles should be preserved. I also tried poking at the software, since it too was open source and asking for contributions.

Somewhere around this time, I became disenchanted with Wikipedia. Perhaps the administrative processes had taken their toll, or maybe I was just tired of arguing. I had never been a good writer (on or offline) so I couldn’t “get back to writing articles” or something of that nature. Facing a void, I turned to Veropedia.

Veropedia was a now-defunt venture by Danny Wool, myself and a few others to showcase the very best of what Wikipedia had to offer. Scraping (yuck) static versions that had been proofread and then using those for display to the end user. Vero was looking for a new developer, so I stepped up. I spent about a year or so working on the project and actually made some decent milestones. I managed to get a Lucene-based search going as well as ported our entire article validation script to PHP from Perl. Our technology was a hodgepodge of scripts, mostly because the original developer thought MediaWiki had too much overhead for our needs. Planning out a MediaWiki-based phase 2 of our software became the new goal.

So I started getting more involved with MediaWiki development. I got commit access, and started working on various bugs that were hindering Veropedia development–sidebar: I originally discovered the libxml2 entity bug while working on some customizations for Special:Import. And so Veropedia kind of just stayed the same while I chugged away at MediaWiki. Somewhere along the line I stopped really caring about Veropedia. I was busy with work, school and now I’d taken up MediaWiki development in my spare time.

Veropedia is dead. At least Veropedia as I’ve known it is dead. Actually, the server I had it living on just got shut down today. I’ve still got the leftover backups lying around, but the site itself is down. Danny still has the domain names if he ever wants to use them again, but I’m done with it. It’s not a wiki.

And that’s really what it’s all about to me at this point. Some point while working on MediaWiki, I realized that that was what I really wanted to do. Work on wikis. Veropedia wasn’t a wiki. Arguments on mailing lists and talkpages aren’t wikis. Collaboratively editing text is what MediaWiki does and what I’m proud to help support. I’ve started working with the Foundation on a contract to support FlaggedRevs/Pending Changes and it’s exciting to make the move from being a volunteer.

I really think that workflow systems like the one I’m helping to support really improve the wiki model. They help pages be as open as possible to editing while still allowing editorial control. And producing free, high-quality content is really what MediaWiki is designed to do. I think it’s pretty cool that I can help make that happen.

Wednesday. The first day was a little rough (I was jetlagged for most of it anyway) I thought. The large groups didn’t work out terribly well and weren’t conducive to getting work done. Although we got a lot of stuff for Google Summer of Code discussed (sidebar: our 2010 projects have been announced. Congrats to the 6 awesome proposals we’re working with this year), there were a lot of other things we needed to talk about too and everyone who attended wasn’t necessarily interested in GSoC. Spent the rest of the day hacking, trying to beat jetlag, and hitting up “Fashion Meets Tech” with Andrew.

Thursday. I have to say the small group structure we adopted on day 2 worked incredibly, and I think we should continue to use that model from now on. Having small groups allowed really focused discussions on things that could really interest you. For example, I was able to work with some people in discussions regarding long-term planning for Bugzilla and Code Review or file metadata–subjects that don’t interest a lot of people–and was able to skip the sessions on OpenID or the Toolserver. It gave you a group of really interested people all very focused on the same task at hand. All of the sessions I attended were fantastic, but I’d like to summarize my two most productive ones below:

New-installer – I led this session in one of the upstairs room just after lunch on Thursday. I was a little hesitant to show it off in the state it’s in, but I have no regrets after doing so. Everyone really seemed to like where it was going, and I got a lot of solid feedback (most of it should be at [[mw:New-installer issues]], feel free to add items if something’s missing). Also began working with Siebrand on the localization aspects. The installer’s i18n has been split into its own file, loaded like an extension, and he completed the Dutch translation so we could start debugging. Roan also took the time to clean up the Javascript and jQueryify it. I think everyone liked it though

Organizational issues – On the first day we mainly talked about GSoC. The second day covered a lot of really important subjects such as bugtracking, code review, release management, the Mediawiki roadmap and volunteer outreach. It was largely a consensus over our known trouble spots and single points of failure. Our new CTO, Danese, informed the group that these issues are known and that specific plans are in place to fill these gaps (contingent on new budgets, et cetera). This is welcome news for a lot of us in Mediawiki/Wikimedia development. The notes of all this are on meta.

Friday. Friday was mostly spent bugsquashing and bouncing ideas off one another. And the day we all started getting very worried about a volcano from Iceland that we couldn’t pronounce. So a bug was filed (and quickly took a life of its own).

The next several days were spent booking and rebooking flights. Changing hotels. Searching for laundromats. Trying to find places for free or cheap wifi. The whole conference wasn’t there (luckily), but the group who were an awesome group of people who made being “stranded in Berlin” a more enjoyable experience. I got to eat great food and talk wikis and tech for days on end. I couldn’t ask to have been stuck with a better group of people. Bonded by our ‘travelupdate’ list and FlightRadar24.com, we managed to all stick it out in Berlin and keep the wikis running.

I’m glad I got to see some familiar faces, some faces I had not seen, and entirely new faces I didn’t know. I won’t be in Gdansk this summer for Wikimania, but hopefully we can manage to pull together the “East Coast Hackfest” for sometime this winter.

IdeaTorrent

People keep kicking around the idea of using IdeaTorrent for tracking enhancements to MediaWiki. I’d like to give it a shot so I’m in the process of setting up an instance of it for the Wikimedia community to play with. Hoping to have this ready by Berlin (if their website will stop going down!).

New-installer

In whatever shape it’s in, I plan to show off the new-installer branch to MediaWiki. This is a huge overhaul of not only the installation process itself, but the backend supporting it. I’ve also managed to sneak a long-overdue schema abstraction in, which should help with database maintenance in the future. I would like to have this in a slightly more workable state by Berlin, but I’ll be showcasing what I have nonetheless.

Git Transition/Code Review/Bug Tracking

People like Git. Some people think we should move to Git and rid ourselves of Subversion. I’m of mixed opinions, but I’m always open to trying new things. I’ve been working with Ævar to get a working Git copy of our Subversion repository with all of the metadata intact (or as much as we can preserve). People also are looking at alternatives to using CodeReview/Bugzilla for our tracking systems for code and bugs/enhancements. I’m hoping to have an instance of Gerrit up and running by Berlin as well, so people can toy with that and see what they think.

A lot has changed in 1.16 (for the brave, the full RELEASE-NOTES), and I’d like to hit on some of the major things here that I think need mention:

• The Metadata editor ($wgUseMetadataEdit) has been split into a separate extension, MetadataEdit
• Introduced CDB interface for high-performance constant data
• Default output format is now HTML5 instead of XHTML 1.0 Transitional (see $wgHtml5)
• Maintenance scripts got a lot of cleanup and reorganization, AdminSettings.php no longer required
• New hooks
• Major improvements in SQLite support
• Test suite is now at least organized (will see more coming here in 1.17)
• Many many other bug fixes and new features

Hopefully we’ll see a general update to the Wikimedia cluster coming in the very near future, meaning that non-critical bugfixes that have been “Fixed in SVN” will finally see production deployment. The first 1.16 release candidate will be coming soon as well, which I encourage those of you using MediaWiki to download and test it out for us. See what works, what needs tweaking and what’s downright broken. It’s a big release, so we’d like to get as much feedback as humanly possible.

Remember that all bugs goto Bugzilla (recently upgraded and restyled), and more information is always better than less.