Build Date: Tue Oct 22 17:50:12 2024 UTC
Divide people up in terms of are they assholes, creeps, or walking corpses.
-- Flesh
Scoundrel: A New Concept For Searching P2P
2001-03-06 01:12:24
One problem still haunts the peer-to-peer (P2P) world: how the hell do you find anything? Traditional search engines are impractical because by the time a P2P network has been spidered, the makeup of the network and the content will probably have changed. Real-time keyword searches are too slow. Yahoo-style index pages don't cut it. We need some crazy new ideas. Enter Scoundrel...
Searching P2P networks sucks. With systems such as Napster and Gnutella you put in keywords, or partial or full titles or artist names, and you get a big list of crap back. There are always tons of redundant results to sift through, and you have to decide which file to download. Once you finally decide on what to snarf, the file transfer bombs out half way through, so you have to try again. This all assumes that you know exactly what you're searching for. Usually there is no cross-indexing.
Then there's the "catch as catch can" problem. Because P2P nodes are connecting and disconnecting to the network at unpredictable times, the availability of resources is constantly in flux. You might be able to find something one day, but the next day the same thing might not be available. If you really want to find something, you may need to search for it over and over again.
Some P2P systems barely have any way to search at all. To discover what's on Freenet, for instance, you generally have to look through big text files full of "keys" (Freenet's version of hyperlinks).
So what can be done about this?
The Scoundrel Project has developed a system for automating the process of searching, re-searching, and downloading things from P2P networks. The core idea is to have what the author of Scoundrel calls a "linkless index." This index, rather than being an index of what actually is on the network, is an index of what theoretically, or ideally, should be on the network. It's "linkless" because the actual index entries do not directly point to resources on the network.
The user can browse this index and select things that he or she would like to retrieve, regardless of whether or not the files are actually available on the network at that particular time. Later on, an agent (bot, or whathaveyou) retrieves the files. The agent does all the shit work of searching, re-searching, selecting the right file, downloading, retrying, etc., all in the background, or while you're away doing something fun. After all, that's what computers should be doing -- tedious drudge work.
This idea has some interesting ramifications. Because the index is disconnected from the actual files on the network, no checks need to be performed to ensure that the index accurately reflects the contents of the network. Thus the index can be more complete, be maintained independently, and kept up-to-date more conveniently. And with a big, comprehensive index, it will be easier to have good cross-referencing.
What's more, the user is free to browse the index at high speeds, selecting things willy-nilly like a kid in a candy store, without waiting for a search to complete, downloads to finish, or being disappointed when a real-time search turns up an empty result set. Everything is ultra-responsive, and it's a better user experience.
Another implication of this strategy is that the network doesn't have to be fast. Some P2P systems are blecherously slow right now. This will certainly change as the software develops and the systems get more users, but at the moment the wait to get a file can be maddening. But who cares if you're not doing all the waiting yourself?
There've also been strange and evil rumblings from the Digital Millennium Copyright Act (DMCA) people about how innocent hyperlinks to net resources containing copyrighted material will be considered some kind of horrible copyright infringement in themselves, punishable by hanging and whatnot. This could put an ugly chill on the whole Internet. A linkless index steps around this issue quite handily.
Although this linkless index strategy can be used for all kinds of data, it is obviously well-suited for digital music trading (e.g. MP3s), which seems to be the focus of many of the P2P projects right now. So to build it's proof-of-concept application, the Scoundrel Project decided to use an existing, highly-developed database for it's linkless index: Amazon.com. Amazon's music index is huge, cross-indexed, chock-full of user reviews, and has all sorts of handy features which make it great for browsing music titles.
Here's how the Scoundrel program works: You fire it up and configure it to know about several OpenNap servers -- the open source clone of the Napster system. Currently, Scoundrel only works with OpenNap. Next, you use Scoundrel's built-in Web browser to navigate Amazon's music section. Scoundrel watches as you browse, and when you visit the description page for a CD, Scoundrel automatically picks up the title, artist, and track listings. You are given an opportunity to review the list of stuff that Scoundrel has created, and to modify and delete items. When you are ready to have Scoundrel go to work for you, you hit the "get'em" button, and it crawls the various OpenNap servers looking for MP3s of the music you want. Then you can minimize Scoundrel and play some Nethack or whatever, or continue to browse Amazon for even more goodies.
This works surprisingly well. As a test, I chose a few CDs from Amazon's "Top Sellers" list, set Scoundrel loose, and went to breakfast. When I came back there were at least two complete CDs in MP3 form on my hard drive, and several partial CDs. And Scoundrel was still out busting ass for me. Anyone who has ever spent all night on Napster trying to put together an entire track list of MP3s knows how cool that is.
Scoundrel isn't perfect, and neither is the linkless index idea. Even though there are copious widgets and screens obstensively indicating what the program is doing, it's hard to figure out. Sometimes it seems to just hang, and sometimes it doesn't seem to search for all of the things you tell it to. But after all, it is just a proof of concept. The author calls it a "technology preview." And while a comprehensive index of music files already exists, and there are other databases for things such as movies (e.g. IMDB), how do we deal with P2P resources that aren't already in a nice tidy index somewhere? And how would you create an index for that stuff?
Another thing is that Scoundrel only runs on Windows. It's an open source project under the GNU GPL, but it's written in some sick language like Delphi (which may be excusable considering that it's only a prototype).
Despite these concerns, I give a big warm Beaujolais to the Scoundrel Project!
There is one last intriguing thing to say about Scoundrel. The author of the program is a mystery man. He remains anonymous to this day. On March 1st, just after releasing the latest incarnation of Scoundrel, he posted a message on the Scoundrel home page announcing that he is abandoning all work on the project, and will never be heard from again, although he hopes that others will continue work on the project. This is from the Scoundrel web page:
Well, so much for what Scoundrel has and has not done. As of today, March 1st, 2001, I will no longer be able to continue development on Scoundrel. I'll be disappearing from the face of the earth and will not be reachable. I will not go into the reasons behind this.
Could it be that the big media companies got to him too? Is the RIAA playing hardball behind the scenes? Will we ever know?
In the meantime, give Scoundrel a whirl.
T O P S T O R I E S
Another Nobel Prize-Winning Author Describes Drunkenness
This book won a Pulitzer Prize. Here's its famous paragraph on getting drunk... (More...)
'Why I'm pretty sure JD Vance had sex with a couch'
True or false? The answers await us in that magical land where all truths are revealed -- the internet. (More...)
In 2010 Dr. Cheng-Huai Ruan discovered a way to cause a patient with an abnormal heartbeat to get back into a normal rhythm by sticking a finger up the patient's ass. (More...)
WKRP in Cincinnati aired from 1978 through 1982. Howard Hesseman played Dr. Johnny Fever, a DJ from Los Angeles who was fired from his previous job for saying the word "booger" on the air. In the show Hesseman would do some dialogue, introduce a song, and start the song. You'd hear a few notes, but never the whole song. (More...)
SF Hippies Can't Get Their Act Together
The annual 420 Hippie Hill event in Golden Gate Park, where large crowds of hippies, wannabe hippies, and hippie poseurs drape themselves in tie dye t-shirts and gather on a hill on 4/20 to smoke weed, was cancelled this year because the organizers couldn't get their act together. (More...)
Mozart to be inducted into the Rock 'n Roll Hall of Fame
Joining such hard-rocking inductees as Abba, Chet Atkins, Nat King Cole, and Neil Diamond, the Rock 'n Roll Hall of Fame is proud to induct Wolfgang Amadeus Mozart. (More...)
C L A S S I C P I G D O G
The Deep Dark Underbelly of the Star Wars Myth, or Ramayana Remembered
It's a fact: Star Wars is a blatant plagiarism of an ancient Asian legend, and the long lines of devout Star Wars freaks are really unscrupulous Asian copyright busters. From Indonesia to Thailand to Nepal, videos are available for sale or rent before they're even released in the US and UK due to this nerdy camcorder-clutching bunch. (More...)
During a magnificent sunny day in a fast receding autumn, the Spock Science Monitor reporters once again blew the playa dust off of their computers and covered the 2002 Burning Man Decompression – held every year just east of Portola Hill in beautiful San Francisco. Both an afternoon and evening issues were released to the unsuspecting crowd of freaks attempting to in some small way experience the euphoria of the playa – if but for a brief afternoon far from the desolation of Northern Nevada. (More...)
Patient Joab's scientifick editorial discusses aspect of the space-time-beer continuum never before processed by sub-bush-robot minds!!! Too fabulantastic to contempulate! (More...)
Johnnie Royale's Guide to Wakes
Wakes can present problems for Bad People of the Future. (If you don't know what a BPotF is, you need to read more of the PDJ.) Sure, your friend is gone and you miss him and that really sucks; it does, I know. But all Bad People of the Future are gonna die, and they have all accepted that fact. They do deserve, however, to have one final kickass party to celebrate all the bad things they've done in the past, present and future. And you, as a friend, have to make sure that their desire for a final send off is well executed (sorry for the pun). That's just the way of BPotFdom. (More...)
A Blast from the Past! Pao Tzu goes over and under the crucial variables in the production and consumption of Salvia Divinorum. A must read for psychonauts of all stripes. (More...)
40 Acres, a Mule, and a Crummy 90-Second Spot on Weekend Update
Consider the plight of the Black Man. The Black Man on "Saturday Night Live," I mean. Has there ever been a more pathetic thing than a token unredeemed for 28 years? Where is the NAACP when you really need them? (More...)