(no subject)

[mdlbear]

Went to a Heather Alexander this evening in Palo Alto. Emmy was reluctant to go, but seemed to enjoy the evening.

Spent about half my day at work bringing our experimental server in Japan up to date, and the other half scheming about caching web servers.

Comments

Caching web servers?

[aerowolf.livejournal.com]

I'm actually curious about caching personal web proxies with search engine capabilities and cache-browsing and -tagging features.

Re: Caching web servers?

[mdlbear]

Last thing I did in that space can be found at http://www.risource.org/PIA/ -- at that time the main purpose was maintaining a common history across multiple browsers. It also had a cache, and the plan was to eventually use it for searching and for figuring out what changes had been made in a page since the last time it was visited.

One could also write other little "agents" that tapped into the request-and-response stream and could do things like filtering. The ad blocker was popular.

I've since stopped using it because the performance was pretty bad -- it was written in Java.

Re: Caching web servers?

[aerowolf.livejournal.com]

Aye... Java is a not-wonderful daemon language.

But what I'm interested in is, for example, a means of figuring out what resources have been viewed, as well as the history of those resources. (Among other things, for academic and job-related research.) To be able to see the tree of browsing habits (using the referer string), as well as cache the data on the page for X amount of time until it's deemed unimportant enough to trash...

Thanks for the pointer!

Re: Caching web servers?

[mdlbear]

Yes, you can do all that pretty easily with a caching proxy. If you want decent performance and total standards compliance, I'd recommend hacking mod_proxy and mod_cache in Apache -- those were the problems that always plagued the PIA.

There are also security issues if your cache is visible to anyone but the user. Not so bad if it's just for research.

Re: Caching web servers?

[aerowolf.livejournal.com]

Isn't that what proxy authorization is for? ;)

My understanding is that there's a sourceforge project to take the PIA and turn it into an apache module, written in C?

(I was thinking about trying to abuse squid's caching and logging mechanism, but... bleh.)

Yay, caching (web) servers!

[aerowolf.livejournal.com]

I've managed to find Delegate (http://www.delegate.org/). Have you had any experience with it?

Re: Yay, caching (web) servers!

[mdlbear]

Looks interesting, but no. Last couple of proxies I worked on were stand-alone (Perl and Java); the first one was based on the CERN daemon (it was a long time ago).

At this point, I'm mostly interested in Apache modules; you could probably do practically anything by starting with mod_perl. But I like Delegate's multi-protocol capabilities; that might prove useful at some point.

Re: Yay, caching (web) servers!

[aerowolf.livejournal.com]

Not only multi-protocol, but almost transparent multi-point protocol. (Say you work for a company that needs to give access to another company to a specific webserver, that that other company has a network layout that makes it impossible to completely and cleanly grant access directly to your webserver. Instead of fretting and forcing a severe hassle on the other company's side, just install Delegate in a couple of spots in the network -- it'll know where to forward to, and a forwarding proxy can just forward that on to the next proxy in the chain fairly easily. As a bonus, you don't even have to open your firewall all that much; just run Delegate on a DMZ box and punch a single hole to that single webserver from that single Delegate server. Use Delegate's access control capabilities to ensure that only a single machine on the remote side can access it, and only via Delegate's proxying protocol... and then have the Delegate on the remote side concentrate all of the individual proxies that needed to be set up within the other company's network.)

I've actually been looking at alternatives to Apache -- yes, it's a wonderful, general-purpose webserver, but there are certain circumstances where it doesn't make any sense. http://www.acme.com/ has thttpd, which is faster for static content. It's also got a tiny SSL-enabled webserver available.

It all depends what you want and need to do what tool to use for the job -- don't go with "the most general-purpose tool", or else you'll find yourself with bloatware and more processing load. :) [I'm sure you know all this -- but it doesn't hurt to remember it.]

Re: Yay, caching (web) servers!

[mdlbear]

thttpd is pretty cool; I also like the throttling feature. I've also written a small server in Java which is seeing some use around the lab in things like PDAs and appliances -- I'll see whether I can release it.

For a general-purpose web server it's hard (impossible?) to beat Apache, but as you say there's a lot to be said for special-purpose servers when they make sense. Sure, you could run Apache on a Zaurus (it's been done), but...