voidstern

Text Extraction

Lukas

·

·

Fiery Feeds 2.13.0 brings big improvements to its text parser and an easy(-ish) way to create custom extraction rules.

Text Extraction

TL/DR: Text Extraction works with more sites, and can handle (some) multi-page sites, and (some) sites that rely on javascript to load content now.

But to get a bit more technical: Ever since Fiery Feeds 2.11.0 over a year ago, text extractions happens on device using a custom, native Swift extraction framework, which is inspired by, and uses the site config rules as FiveFilters, and Instapaper before it.

Before this release, Fiery Feeds only supported the most important subset of possible rules (XPath to body, title, date, author, and stripping elements), but 2.13.0 adds support for a couple more rules, that are supported by FiveFilters, but used less than the

  • single_page_link and next_page_link – if those links are present on the site, and the rule is set, it can combine multipage articles into a single text
  • replace_string([string to find]): [replacement string] – can be used to replace parts of the html before trying to parse and extract the content.
  • strip_image_src – can remove certain images, mostly ads.

I’ve also added an option, that is not supported by FiveFilters, but I think will come in handy, since a few sites have started to only provide a basic template and load the actual article content afterwards using javascript.

  • web_view_load – if set to yes Fiery Feeds will not use the standard HTTP calls to fetch the content, but load the URL in a hidden webview, wait until the site has finished and then fetch the HTML from the web view, after any javascript was executed.

Additional I’ve improved the fallback logic, if there are no rules for a given site, and if this fails as well, there is now a second, boilerpipe-style fallback algorithm, which doesn’t support images, unfortunately, but I have yet to see it site where it fails to get the text content.

Site Config

The additional rules above are used in the rule set that is shipped with Fiery Feeds, but 2.13.0 also adds an iCloud folder where you can put custom rules.

To make it easier to create the custom rules files, I’ve created – mostly for myself, admittedly – a Mac App called Site Config. (That’s the reason why I went with an iCloud folder for custom rules – it makes it easy to create the rule on your Mac, put it in said folder, and Fiery Feeds on iPhone will just pick it up.)

It has a web view with a point a click interface to select the important elements (title, body, author, …), text editor to edit the rules file directly, and most importantly a (somewhat) live preview using the same extraction framework, that also powers Fiery Feeds.

It’s still fairly rudimentary, and I expect to improve especially the UI over the coming months, but it’s already making my life a lot easier, so I figured I might as well share it with the world.