Category Archives: http

>battling the parser!

Posted on by

>Today I finished up the refactoring that was needed to remove all the blocking code from my POE app.  I hate that they were in there to start with, but it looks good now.  I did have a bit of a fight with the web client module I swapped to, POE::Declare::HTTP::Client.  Apparently the old client module I was using had it’s own implementation of a HTTP parser, but this one uses HTTP::Parser, how original.  :)  I thought that was a good thing until it died from a bad header.  Checked the old client and it would take junk headers and relist them with the name Client-Junk: XXXX (replacing XXXX with the malformed header that was there to start with).  I’ve submitted a bug request to the web service I’m calling that has the malformed header, but how should I proceed?

At first I was going to submit a bug request to HTTP::Parser.  I looked into the code, found the line where it was dying because of a malformed header, and changed it so it didn’t die.  I mean, why should my application die over the format of content that I’m not in control of?  Sure, give me a warning, but don’t die.  Well, the more I thought about it, two things got to me about this logic.  First, this module has been around a long time behaving this way and there are numerous other places where the module dies.  It’s kind of fix them all or leave them be.  Second, Paul Finwick said it best with his Ferrengi talk at the past couple YAPC conferences.  Tis better to die than to return in failure.  Wrap that little shit in a try block sir :)

I went to the POE::Declare::HTTP::Client module, dug around and saw that it’s being accessed from POE::Filter::HTTP::Parser.  Wait, it’s already in an eval block.  Why the hell isn’t that catching the die?!?

*sigh* Maybe tomorrow I’ll figure it out. :)  Goodnight.

Pin It

>refactoring the ua requests

Posted on by

>Currently a system I’m working on takes HTTP requests that it needs to send, but doesn’t really care much about the response, and tosses them over to a POE HTTP client that just sends in the background. The problem here is that it does each request concurrently. Normally that’s not an issue at all, but as it turns out if the web server you are sending these too gets overloaded and starts taking a long time to respond to request, all of a sudden you are driving up the load on my app server because of how many backlogged requests I’ve got danging concurrently.

Let’s find a solution for this.  Not a big deal, I don’t really care about the requests much so let’s just queue them up.  So rather than having the generic http ua session tossing things like wild, wrote up a quick WebClient module that maintains a queue of requests and handles them.  Since some requests are more important to others, let’s use a priority queue.  Simple to implement, but POE::Queue comes out of the box since we are already using POE, so we go with that.  For the HTTP client, I’m trying out POE::Declare::HTTP::Client.  I’ve had some issues with the generic component http client, so we’ll see if we fare any better with this one.

So…

Simple API. *check*
Queued requests to prevent overloading. *check*
Priority aware to take care of important things first. *check*
Optional callbacks in case we do care about the response. *check*

Sounds like a winner.  Writing up some test cases now, but looks good and functional.  Thanks CPAN for making this pretty damn easy to throw together. ;)

Pin It