I get questions almost every day from publishers asking how they can improve the way Instapaper parses their sites. And I get almost as many emails from web programmers among Instapaper’s userbase asking if they can contribute custom-parsing instructions for their favorite sites.
Well, I’ve been working on a solution to this for a while, and it’s almost done.
Part 1: Customized parsing
The Instapaper text parser has a complex “automatic” mode in which it tries to make its best guess on which element in any given page is the “body” container that will include all article text with minimal clutter from non-text elements (e.g. navigation, headers, comment forms, share-this widgets, etc.) and which additional elements inside the body container should be removed.
In addition to the automatic behavior, I can custom-configure the parser’s behavior for specific sites. So, for example, if a popular site such as Los Angeles Times parses poorly under the automatic behavior, I can customize the parsing of all
latimes.com stories with fine-tuned directives:
body_node = //div[@id = 'story'] strip_id_or_class_substring: 'related' strip_id_or_class_substring: 'tools'
But I have very little time to keep this list updated, and if a site’s not a major Instapaper source, I’ll probably never get around to it. This, obviously, sucks. But there’s a better way.
Starting very soon, I’ll open these up for public contributions. If you know a bit of XPath, you’ll be able to test and submit custom parsing instructions for any site. They’ll go through admin approval before going live, to prevent abuse, and then they’ll improve the text parser for everyone.
And, recognizing that your efforts could be useful to a wide range of other tools and services, I’ll make the list of all of these site-specific configurations available to the public, free, with no strings attached.
Sure, someone could make a pretty great competitor to my text parser with this list, but I’d rather have a better common parsing database at the expense of exclusivity instead of trying to keep this one component to myself at the expense of its quality.
Part 2: Special class names
Additionally, the Instapaper text parser will support some standard CSS class names to instruct it:
instapaper_body: This element is the body container.
instapaper_ignore: These elements, when inside the body container, should be removed from the text parser’s output.
instapaper_body class name is present, the automatic selection and automatic stripping processes are disabled, leaving full control to the site’s author.
The back-end work for this is almost entirely done, and the interface is about half-done. (The submissions mechanism needs a lot of interface work.)
I expect this to be live within a few weeks. Until then, if you have any feedback or would like to see any other special class names added, I’d love to hear from you: email firstname.lastname@example.org. Thanks.