A (hopefully) final update on the FBI raid

I’ve had a chance to correspond with DigitalOne over the last few days to clarify what happened with the server raid last week in which the FBI (seemingly accidentally) took one of Instapaper’s database servers and numerous other unrelated servers when attempting to seize another server allegedly used for malware distribution.

I initially assumed and published that when the FBI took my server, they had possession of the hard disks and therefore a copy of the Instapaper database and source code.

DigitalOne’s CEO, Sergei Arsentiev, informed me in an email last night that the FBI agents only took one enclosure containing 16 HP C7000 blade servers (including the one I was leasing), but that my server’s two hard drives were in a separate HP MDS600 enclosure that was not taken, and the drives were never removed from it.

If this is accurate, the FBI did not take my server’s hard drives from the datacenter or their enclosure — they only took the C7000 module that contained my server’s processor, RAM, and other non-storage components.

I don’t know enough about the physical workings to say for sure, and I wasn’t present to observe the agents’ actions, but this makes it very unlikely that they read or copied any of Instapaper’s data or code.

I’m relieved to learn this, and I’m considering the matter closed for now. Thank you, everyone, for your support during this ordeal.

Server update

About 40 minutes ago, at 11:45 PM EST on June 23, the server that I believe the FBI took during the DigitalOne raid came back online for the first time.

The logs indicate that it was not booted into its OS during the time it was missing.

DigitalOne has still said nothing about the raid to me, but my best guess is that the FBI did have physical possession of the server for this period. If so, they could have copied the data from the drives without booting from them and without leaving evidence of the copy.

Since it was returned so quickly, it’s likely that they determined that it wasn’t part of their target group and wanted to avoid any problems that could have resulted from its continued seizure. While they could have copied the data for future analysis, I believe it’s unlikely that they would have reason to do so. Regardless, I have no way to know what they did (or didn’t do) with it.

For whatever it’s worth, I have deleted the code, data, and keys from the server and asked DigitalOne to cancel my account immediately. I’m not convinced that they did everything they could to prevent the seizure of non-targeted servers, and their lack of proactive communication with the affected customers is beneath the level of service I expect from a host.

Many commenters and emailers have taught me that bcrypt and scrypt are better than salted SHA-1 hashes for password storage, so I’m researching them and will begin load-testing with them next week. If all goes well, I’ll deploy one of them and migrate all subsequent logins and password changes away from salted SHA-1 hashes.

I appreciate the outreach from people wanting to help me fight the FBI or DigitalOne somehow, but that’s honestly the last thing I’d want to do. Even if money were no object, I can’t afford the time or the stress, I’m not looking for any sort of reimbursement, and nothing they say would absolutely assure me (or even the slightest skeptics) that they had zero copies of the data.

I have a great product to maintain, expand, and improve, and there’s nothing I’d rather do than get back to work doing what I love.

The FBI stole an Instapaper server in an unrelated raid

UPDATE: The server has been returned. Please read.

One of Instapaper’s five leased servers was hosted at DigitalOne, a Swiss hosting company leasing blade servers from a Virginia datacenter. Early Tuesday morning, the FBI raided the datacenter to seize servers used by another DigitalOne customer for fraudulent “scareware” distribution, according to the FBI’s press release, but they seemingly took a lot more servers that happened to be physically near the server(s) they were looking for.

There’s very little information on this, but The New York Times has the most complete coverage in Tuesday’s Bits post:

The F.B.I. seized Web servers in a raid on a data center early Tuesday, causing several Web sites, including those run by the New York publisher Curbed Network, to go offline. …

In an e-mail to one of its clients on Tuesday afternoon, DigitalOne’s chief executive, Sergej Ostroumow, said: “This problem is caused by the F.B.I., not our company. In the night F.B.I. has taken 3 enclosures with equipment plugged into them, possibly including your server — we cannot check it.”

Mr. Ostroumow said that the F.B.I. was only interested in one of the company’s clients but had taken servers used by “tens of clients.”

The LA Times also has good coverage:

“FBI was interested in one of our clients and in his servers, but they took besides target servers tens of not related servers of other customers,” [Ostroumow] said.

As far as I know, my single DigitalOne server was among those taken by the FBI (which I’m now calling “stolen” since I assume it was not included in the warrant). I’m assuming this because it became unreachable and stopped sending updates to my internal monitoring system at approximately the time that the FBI raided the datacenter, and has not come online again since then.

The server was used as a MySQL replication slave, handling read-only queries to speed up the site. Instapaper suffered no downtime as a result of its theft and no data has been lost, but site performance has been slower without it.

Instapaper’s main host, SoftLayer, responded quickly to an order I placed to replace this server there. It’s almost completely set up, and the site’s performance should be fully restored by tonight.

What the FBI stole from Instapaper

I didn’t own the hardware — I was leasing it from DigitalOne. So the FBI has only stolen my time and a partial month of hosting fees, not any physical property of mine. (The hardware was pretty expensive to DigitalOne, though: each of these servers probably costs $5,000–8,000.)

Possibly most importantly, though, the FBI is now presumably in possession of a complete copy of the Instapaper database as it stood on Tuesday morning, including the complete list of users and any non-deleted bookmarks. (“Archived” bookmarks are not deleted. “Deleted” bookmarks are hard-deleted out of the database immediately.)

Instapaper stores only salted SHA-1 hashes of passwords, so those are relatively safe. But email addresses are stored in the clear, as is the saved content of each bookmark saved by the bookmarklet.

The server also contained a complete copy of the Instapaper website codebase, but not the codebase of the iOS app.

Linked Facebook, Twitter, or Tumblr accounts only store their respective OAuth keys. Linked Evernote accounts only store the Evernote email-in address. Linked Pinboard accounts, however, store plaintext usernames and encrypted passwords, and the encryption keys are present in the website source code on the server.

So the FBI now has illegal possession of nearly all of Instapaper’s data and a moderate portion of its codebase, and as far as I know, this is completely out of my control.

Due to the police culture in the United States, especially at the federal level, I don’t expect to ever get an explanation for this, have the server or its data returned, or be reimbursed for the damage they have illegally caused.

UPDATE: Fortunately, I was wrong about one of those: The server has been returned. Please read.

I’m really not sure what to do about this. I’m speaking to my lawyer about it shortly, but as far as I know, there’s nothing I can reasonably do without spending more money, time, and stress than I can afford on a path that would likely lead nowhere productive.

DigitalOne’s response

DigitalOne hasn’t handled this well. Nobody from the company has contacted me at all since this began. The company’s website is still down, suggesting that they might not have any backups (or at least don’t care enough) to set up a temporary page elsewhere. I have no idea whether I’ll ever see the server again, whether I’ll be reimbursed for the remainder of the month that I’m not receiving the service I paid for, or whether I’ll be billed on July 1 for the next month of nonexisting service.

It’s also possible that miscommunication or a lack of communication from DigitalOne caused the FBI to be so imprecise in what they took. Nobody really knows except DigitalOne and the FBI, and neither are being particularly helpful.

Regardless, I’m not hosting any servers at DigitalOne in the future, and I’m not renewing this one (if that ever becomes possible).

UPDATE: The server has been returned. Please read.

Mini-FAQ on iOS updates, Reading List, iCloud

A lot of people have asked these questions on Twitter or via email since the WWDC keynote last week, so I’ve consolidated my answers here:

Will Safari’s Reading List feature “kill” Instapaper?

I highly doubt it.

But I’m not standing still. I’m preparing some great updates for the coming months.

Can Instapaper integrate with Reading List somehow?

No. As far as I know, there’s no API to access Reading List’s contents or add pages to it.

Does Instapaper still support iOS 3.1.3?

Today: Yes. In the near future: No.

There are a number of great features that I want to use in iOS 4 and 5, and compatibility with iOS 3.1.3 is starting to become an expensive burden. My code is quickly filling with special cases and duplicate implementations, and this will hinder Instapaper’s feature development, maintenance, and stability until I can eliminate 3.1.3 compatibility. My quality assurance would also improve since my testing-device count would be greatly reduced.

So in one of the next releases, I’m raising the minimum to iOS 4.0. This will drop compatibility for the original iPhone and first-generation iPod Touch, all of which are now at least three years old and represent only 0.48% of Instapaper’s customers.

And in the not-too-distant future, possibly as soon as next spring, I intend to raise the minimum to iOS 5 because it has tons of great UIKit improvements and opportunities for me to delete lots of hacky legacy code. iOS 5 excludes the iPhone 3G and second-generation iPod Touch as well, which represent 3.4% of customers today, but that number is falling quickly.

Can Instapaper use Newsstand to get automatic background downloading?

Probably not. The specifics are under the WWDC-attendee NDA, but I think I’m safe to say that only Newsstand apps get access to background downloads, and there would be a number of downsides to making Instapaper a Newsstand app.

It’s also unclear whether this sort of use would be rejected. And since it might be right on the edge of permissable, it’s also a high risk that it might be approved and then later pulled from sale, which would potentially be very expensive.

I want to wait and see how other non-publication apps, such as feed readers, handle Newsstand before I’ll consider it. But it’s probably smarter to wait for Apple to possibly give more apps automatic-background-download abilities in the future.

Can Instapaper use iCloud to simulate automatic background downloading?

Not really. The idea would be to have an always-running Mac app on each customer’s computer to push newly saved documents through iCloud and onto the devices. But Apple doesn’t want developers storing this type of data in iCloud (the specifics of which are, again, probably under the WWDC NDA), and I don’t want to risk rejection.

Can Instapaper move its service to iCloud and get rid of its servers?

No. This could only work if Instapaper was just the isolated iOS app. But there’s an entire web service backing it that gives it benefits that iCloud can’t, such as support for all web browsers, non-Apple devices (most notably including the Kindle), a dynamic page-transformation engine, and API additions from hundreds of other services and apps.

Can Instapaper use the built-in iOS 5 dictionary and drop its current one from the download?

Possibly. Instapaper’s built-in dictionary has some useful features that Apple’s dictionary doesn’t offer, including words that link to their own definitions and support for other installed dictionary apps such as Terminology and foreign-language dictionaries.

So I might be able to use Apple’s exclusively in the future if it gets some improvements, but for now, I’ll probably just make it an option and continue to bundle my own, which only requires about 16 MB of space on the device.

Can Instapaper switch to the new iOS 5 Twitter integration?

Not without losing significant functionality, such as offline queuing of tweets. For now, I don’t plan to switch to it, although I may offer it as an alternative.

Can Instapaper use the new hardware-brightness control in iOS 5?

Yes! Can’t wait.

Can Instapaper use the iBooks-like page-turning mechanics in iOS 5?

Probably. I’d like to, but it would require a restructuring of my pagination code, and I’m not sure it’s reasonably possible for Instapaper’s content and considerations. I’ll be experimenting with this a lot in the next few months to see if I can get it working without significant drawbacks.

Instapaper 3.0.3 update

This is mostly a bugfix update:

• Fixed bugs with pagination, tilt-scrolling, AirPrint, and other minor issues.
• Consolidated search bar to always search all titles and content.

• Added support for more dictionary apps in the Define window. The full list is now:

  • Terminology for iPad and iPhone
  • American Heritage Deluxe
  • American Heritage Desk
  • Collins Gem
  • Oxford Deluxe
  • Australian Oxford
  • Longman English-to-Japanese
  • Longman Dictionary of Contemporary English (LDOCE5)
  • Collins Gem Mandarin Chinese-to-English
  • Collins Gem French-to-English
  • Collins Gem Malay-to-English

  Each app is only displayed if you have it installed.

Get it now, and of course, it’s free to all existing customers.

Instapaper 3.0.2 for iPhone and iPad now available

tl;dr version: New dictionary, zoomable images, lots of bugfixes. Get it.

New dictionary

Did you know that Instapaper has a built-in, offline English dictionary? (It takes up most of the app’s 10 MB download size.) It’s been there for a while, but in 3.0.2, it got a huge overhaul with lots of improvements.

Just select a word and tap Define:

In 3.0.2, this has an all-new look, thousands of new and expanded definitions, and interlinking between words. So in this case, in the relatively unhelpful definition of “Of, or pertaining to, the Apocrypha”, you can tap “Apocrypha” and see what that means.

And if you have the excellent Terminology (iPad, iPhone) app installed, that link in the corner appears automatically, and you can continue your search there if you’d like to explore further.

Zoom images

Tap any image in an article, and you now get a helpful menu, including a “Zoom” option to open them in a full-screen, zoomable image viewer:

(Hi, Jim!)

For a shortcut, just pinch to zoom on any image and it opens in the Zoom viewer.

Developers: x-callback-url

For the dictionary look-up-in-Terminology feature, Instapaper and Terminology communicate via the new x-callback-url draft standard that we co-authored (in that I had a vague idea, then Greg Pierce, author of Terminology, did all of the actual work and documentation).

Using this, Instapaper calls out to Terminology, but then Terminology offers users a quick way to switch back to Instapaper when they’re done.

Instapaper now supports its own x-callback-url method for adding URLs, in case you want to skip the Instapaper API or provide offline adding-to-Instapaper without implementing a queue:

x-callback-instapaper://x-callback-url/add?…
Parameters:

• url: (required) The URL to add.
• x-source: (optional) The human-readable name of your app that will be presented in the dialog (e.g. “Add this URL from {source}?”)
• x-success: (optional) The URL to call back after the user has responded to the Instapaper “Add” dialog. If unspecified, Instapaper will remain open.
• x-error: (optional) The URL to call back if it fails for some unforeseen reason or if the user isn’t logged in. If unspecified, the x-success URL is called.

Bugfixes

The 3.0.2 release fixes a lot of bugs, big and small. Among the notable ones:

• A memory leak that would cause occasional crashes.
• The bug in 3.0.1 that caused the most recent article not to show up after an update if it contained images until after you hit Update again.
• Pagination preventing the device from sleeping forever.

Get the update now in the App Store, free for all customers and $4.99 if you aren’t a customer yet.

Instapaper 3.0 is here!

Instapaper 3.0 for iPhone and iPad is now available in the App Store, and I’ve updated the Instapaper website to include many of its new features as well. This is the biggest update Instapaper has ever had in one version, and I’m proud to finally show it to you.

Sharing

Both the app and the website now have full-featured, native sharing to Facebook, Twitter, Tumblr, Pinboard, and Evernote.

Your credentials and sharing preferences sync between the website and the app, so you only need to log into each service from one place.

You can post to any of these services from the app offline, and the app will queue up the post to be submitted next time it’s online.

Like prior versions, Tumblr posts can be created as Draft, Queued, or Published posts on any of your account’s Tumblr blogs, and if you have any text selected when you hit the Share button in the app, it will offer the choice of creating a Quote or Link post. (This is how I create almost every quote on my blog: with Instapaper set to create Draft posts, I quote interesting selections from articles and post them, and then I filter through them later from my computer to add commentary and publish the best ones.)

You can link your accounts on these services to Instapaper on the Account page, or in the app’s Settings panel under “Sharing Accounts”.

Stars are now Likes

Nobody knew when or why to use Stars, so I’ve renamed them to Likes to clarify their purpose. Generally, you should Like articles that you think are interesting and that you might recommend to others. Because…

Browse Likes from friends

Instapaper is now social! (Don’t worry, it’s tasteful and optional.)

You can now browse your friends’ Liked items to find great articles to read.

You can find friends through your linked Facebook or Twitter accounts, your email addresses in Contacts (from the app), or by entering their email addresses yourself (on the website). Since we’re starting from zero, check back often over the coming weeks to find more friends who linked their Facebook or Twitter accounts with Instapaper.

You can see your friends’ Liked articles and add more friends from the Browse section on the website, or the Friends button in the app.

Don’t forget to link your Facebook and/or Twitter accounts and start Liking articles when they’re good!

Browse editors’ picks, or any website

The app also now has a completely rewritten web browser built in, so you can browse to any website, find the articles you want to read later, and save them directly from the app.

And there’s a new Editors browser, featuring the best human curators on the web who recommend great articles for Instapaper reading.

With these great new additions, many customers won’t even need the bookmarklet anymore.

Phasing out RSS folders

With all of these great new ways to find pages to read, I’ve decided to phase out the rarely-used RSS-folder feature. In 3.0, it’s no longer possible to create new RSS folders, and existing RSS folders will stop being updated soon.

This results in much faster updates and downloads, far less space usage on your iPhone and iPad, and much less bandwidth usage over your 3G connections. I previously had a handful of RSS folders on my account, but I never used them — and my updates absolutely fly without them now. Without this burden, I’ll also be able to increase the number of Starred and Archived items retained by the app (optionally, of course) in the near future.

More new 3.0 features

In addition to the great new sharing and browsing systems, 3.0 has a number of major improvements:

• New storage engine with perfect image quality, faster downloads, faster page-opening, and less space usage. To get all of the images properly and write the new data, all articles will be re-downloaded after updating. Be sure you update when you have enough time to download them all. (Fortunately, it’s a lot faster than before!)
• The app can now search the content of all of its downloaded articles. It’s not website-powered yet, so it can’t search your entire online Archive, but I’m looking into that for the future.
• The new “smart rotation lock” displays an overlay with a Rotation switch after you rotate, but to avoid annoying you constantly, it only appears if the rotation seemed accidental. (If you turn Rotation off, you can turn it back on at any time from the Settings pane.)
• The Share panel now supports creating tasks in Things.
• The Settings pane now has two new advanced preferences: Reverse tilt-scrolling direction, and disable helpful dialogs. (If you really want to disable the scroll-to-top protection or the scroll-during-pagination dialog, this is for you.)
• The iPad no longer uses the top and bottom gradient to fade the text out, instead opting for a clean shadow line borrowed from Reeder’s style. I’d like to thank Silvio Rizzi, Reeder’s author, for generously giving me permission to copy this great design element.

Plus many more big and small fixes and improvements.

Go get it

As usual, Instapaper 3.0 is a free upgrade to anyone who has ever bought the paid app. For new customers, it’s only $4.99 for the universal iPhone and iPad app. What are you waiting for?

Get it now!

Full API now available

The full Instapaper API is now available for developers. See the Full API documentation to get started.

I’ve made an unusual decision for it that I’d like to explain. I was reluctant to make a full API before now for a few reasons:

• Instapaper has nontrivial operating costs. If a large number of people used it exclusively via someone else’s API app, and never saw the Instapaper website’s ads or purchased the Instapaper iOS app, I’d lose a lot of money supporting those users.
• One of the biggest uses of a full API is to make mobile apps. But Instapaper already has an iOS app, and this generates nearly all of its income. If someone else’s Instapaper app significantly reduced its sales, or forced it to become free to retain a significant userbase, the service might not survive.

Services with venture-capital backing can keep their APIs free in order to get more users, delaying theoretical profitability to an unspecified future date. For a lot of reasons beyond the scope of this post, that’s not the path I’ve chosen for Instapaper. Since it’s a “normal” business that needs to remain profitable to keep operating, it can’t lose money on a very large number of users indefinitely.

Initially, I came up with two options:

• I could limit the API so it couldn’t be used to make a full-featured Instapaper app. This is the method I’ve implemented before today. But people have evaded this by creating unofficial, fragile apps that scrape the website for data. Such hacks cause problems for everyone.
• I could allow full API access only to developers who agree to pay Instapaper a portion of each app’s sale price. But this is legally and financially more complex and time-consuming than it’s worth for most developers (and for Instapaper), and it imposes a lot of constraints.

Those options aren’t very good. But recently, I came up with a third option that I think is much better:

• Full API access, but only for paid-subscriber accounts. In other words, all developers can use the Full API, but it will only work for customers with Instapaper’s $1/month Subscription memberships.

All previous API functionality will remain free and will work for any account. These methods can only add pages to people’s Instapaper accounts, as implemented in many feed readers and Twitter clients. This part of the API is not changing, and it’s now called the Simple API. To clarify:

• If you have already implemented the Instapaper API in your app, you don’t need to change anything.
• If you’re writing support for Instapaper, and all you need to do is add pages to someone’s Instapaper account, they don’t need to be a Subscriber and you can use the Simple API.
• If your app needs to read data from an Instapaper account, it must use the new Full API, and its customers need to be Subscribers.

Instapaper’s own paid and ad-supported free iPhone and iPad apps in the App Store will continue to work for all customers.

The first app

The first complete Full API app is Stacks for Instapaper, a Windows Phone 7 client. I’d like to thank its developer, Matthieu Guyonnet-Duluc, for helping me find and fix issues with the API. (I’d like to thank the other developers as well, but their apps aren’t finished yet, and I don’t want to blow their cover.)

OAuth 2.0

This version of the Full API requires OAuth 1.0 with xAuth. I’m investigating support for OAuth 2.0 if there’s enough demand. If you’d like to see it, please let me know.

Thank you

Finally, thank you all, developers and customers, for showing so much interest in Instapaper that a full-featured API has become one of its most popular feature requests.