Getting full contents for partial feeds

In my opinion partial feeds are not feeds. While I understand the need to get pageviews, I don’t like it. My time is valuable and I don’t want to hop between Google Reader and other browser windows to read the content. Disclaimer: this method might or might not be a violation of some laws, TOS, etc. IANAL. Use this method at your own risk.

The method: use Yahoo! Pipes to fetch the HTML page for each entry. The setup can be seen below:

The feed used in the example is the old feed from the Truested Source blog (since they’ve been bought by McAfee, they publish a new feed with the complete posts ;-)). The first operator (the Loop) fetches each page specified in the link for the element. Two remarks:

• site owners can prohibit Yahoo Pipes from fetching pages using robots.txt
• pages are not fetched at each evaluation of the pipe, rather at each change of the source feed (for those who are worried about a pipe DDoS-ing the site)

Set the “Cut content from” and “to” so that what you obtain the HTML part you want. “Split using delimiter” must be set to something, preferably something which doesn’t occur in the text. I just used some random MD5.

The second loop tries to protect against XSS-ing yourself :-). I discovered this by accident, because the feed contained the following post: A Little Filtering Can Halt Some XSS Attacks. The problem is that the inserted HTML content gets double decoded, resulting in execution of the script, even if it was encoded properly for the HTML page. The method used in the above example is rather lame, hower there is a good news: Google Reader disallows Javascript so you are not at risk, even without this transformation.

Enjoy your full feeds.

Update: Originally I came up with the idea while reading the article Build a Web Page Monitor with Google Docs and Track Changes Automatically, however the Yahoo Pipes solution is much cleaner, task oriented solution (but the Google Docs one is still worth checking out for other possible usecases).

Update: two alternative solutions (which are easier to use than creating a custom pipe for every feed) - via taint.org:

• fivefilters.org
• echodittolabs.org