[NEC] 2.10: File-sharing Goes Social

[list-replies@shirky.com]

NEC @ Shirky.com, a mailing list about Networks, Economics, and Culture 

           Published periodically / #2.10 / October 12, 2003 
               Subscribe at http://shirky.com/nec.html
			       Archived at http://shirky.com
          Social Software weblog at http://corante.com/many/

In this issue:

 - Introduction
 - Essay: File-sharing Goes Social
     Also at http://www.shirky.com/writings/file-sharing_social.html
 - Worth Reading:
   Towsend on telecom and cities

* Introduction =======================================================

This month's essay concerns the way the RIAA is creating envrionmental
pressures that alter the design of file sharing networks, and how the
current attacks on Kazaa et al are moving file sharing into socially
bounded cells.

-clay

* Essay ==============================================================

File-sharing Goes Social
  http://www.shirky.com/writings/file-sharing_social.html

The RIAA has taken us on a tour of networking strategies in the last
few years, by constantly changing the environment file-sharing systems
operate in. In hostile environments, organisms often adapt to become
less energetic but harder to kill, and so it is now. With the RIAA's
waves of legal attacks driving experimentation with decentralized
file-sharing tools, file-sharing networks have progressively traded
efficiency for resistance to legal attack. 

The RIAA has slowly altered the environment so that relatively
efficient systems like Napster were killed, opening up a niche for
more decentralized systems like Gnutella and Kazaa. With their current
campaign against Kazaa in full swing, we are about to see another
shift in network design, one that will have file sharers adopting
tools originally designed for secure collaboration in a corporate
setting.

Napster's problem, of course, was that although Napster nodes acted as
both client and server, the central database still gave the RIAA a
single target. Seeing this, Gnutella and Kazaa shifted to a mesh of
nodes that could each act as client, server, and router. These
networks are self-assembling and self-reconfiguring with a minimum of
bootstrapping, and decentralize even addresses and pointers to files. 

The RIAA is now attacking these networks using a strategy that could
be called Crush the Connectors. A number of recent books on networks,
such as Gladwell's _The Tipping Point_ [isbn.nu/0316316962],
Barabasi's _Linked_ [isbn.nu/0738206679], and Watts' _Six Degrees_
[isbn.nu/0393041425], have noted that large, loosely connected
networks derive their effectiveness from a small number of highly
connected nodes, a pattern called a Small World network. As a result,
random attacks, even massive ones, typically leave the network only
modestly damaged.

The flipside is that attacks that specifically target the most
connected nodes are disproportionately effective. The RIAA's Crush the
Connectors strategy will work, not simply because highly publicized
legal action will deter some users, but because the value of the
system will decay badly if the RIAA succeeds in removing even a small
number of the best-provisioned nodes.

However, it will not work as well as the RIAA wants, even ignoring the
public relations fallout, for two reasons. The first is that combining
client, server, and router in one piece of software is not the last
move available to network designers -- there is still the firewall.
And the second is simply the math of popular music -- there are more
people than songs.

- Networks, Horizons, and Membranes

Napster was the last file-sharing system that was boundary-less by
design. There was, at least in theory, one Napster universe at any
given moment, and it was globally searchable. Gnutella, Kazaa, and
other similar systems set out to decentralize even the address and
search functions. This made these systems more robust in the face of
legal challenges, but added an internal limit -- the search horizon. 

Since such systems have no central database, they relay requests
through the system from one node to the next. However, the "Ask two
friends to ask two friends ad infinitum" search method can swamp the
system. As a result, these systems usually limit the spread of search
requests, creating an internal horizon. The tradeoff here is between
the value of any given search (deeper searches are more effective) vs
the load on the system as a whole (shallower searches reduce
communications overhead.) In a world where the RIAA's attack mode was
to go after central resources, this tradeoff worked well -- efficient
enough, and resistant to Napster-style lawsuits.

However, these systems are themselves vulnerable in two ways -- first,
anything that reduces the number of songs inside any given user's
search horizon reduces the value of the system, causing some users to
defect, which weakens the system still further. Second, because search
horizons are only perceptual borders, the activity of the whole
network can be observed by a determined attacker running multiple
nodes as observation points. The RIAA is relying on both weaknesses in
its current attack.

By working to remove those users who make a large number of files
persistently available, the RIAA can limit the amount of accessible
music and the trust the average user has in the system. Many of the
early reports on the Crush the Connectors strategy suggest that users
are not just angry with the RIAA, but with Kazaa as well, for failing
to protect them.

The very fact that Crush the Connectors is an attack on
trustworthiness, however, points to one obvious reaction: move from a
system with search horizons to one with real membranes, and making
those membranes social as well as technological.

- Trust as a Border 

There are several activities that are both illegal and popular, and
these suffer from what economists call high transaction costs. Buying
marijuana involves considerably more work than buying roses, in part
because every transaction involves risk for both parties, and in part
because neither party can rely on the courts for redress from unfair
transactions. As a result, the market for marijuana today (or NYC
tattoo artists in the 1980s, or gin in the 1920s, etc) involves
trusted intermediaries who broker introductions. 

These intermediaries act as a kind of social Visa system; in the same
way a credit card issuer has a relationship with both buyer and
seller, and an incentive to see that transactions go well, an
introducer in an illegal transaction has an incentive to make sure
that neither side defects from the transaction. And all parties, of
course, have an incentive to avoid detection.

This is a different kind of border than a search horizon. Instead of
being able to search for resources a certain topological distance from
you, you search for resources a certain social distance from
you. (This is also the guiding principle behind services like LinkedIn
and Friendster, though in practice they represent their user's
networks as being much larger than real-world social boundaries are.)

Such a system would add a firewall of sorts to the client, server, and
router functions of existing systems, and that firewall would serve
two separate but related needs. It would make the shared space
inaccessible to new users without some sort of invitation from
existing users, and it would likewise make all activity inside the
space unobservable to the outside world. 

Though the press is calling such systems "darknets" and intimating
that they are the work of some sort of internet underground, those two
requirements -- controlled membership and encrypted file transfer --
actually describe business needs better than consumer needs.

There are many ways to move to such membrane-bounded systems, of
course, including retrofitting existing networks to allow sub-groups
with controlled membership (possibly using email white-list or IM
buddy-list tools); adopting any of the current peer-to-peer tools
designed for secure collaboration (e.g. Groove [groove.com], Shinkuro
[shinkuro.com], WASTE [jabberwocky.de/waste/], etc); or even going to
physical distribution. As Andrew Odlyzko has pointed out, sending
disks through the mail can move enough bits in a 24 hour period to
qualify as broadband [www.firstmonday.dk/issues/issue8_9/odlyzko/#o6],
and there are now file-sharing networks whose members simply snail
mail one another mountable drives of music.

A critical factor here is the social fabric -- as designers of secure
networks know, protecting the perimeter of a network only works if the
people inside the perimeter are trustworthy. New entrants can only be
let into such a system if they are somehow vetted or vouched for, and
the existing members must have something at stake in the behavior of
the new arrivals. 

The disadvantage of social sharing is simple -- limited membership
means fewer files. The advantage is equally simple -- a socially
bounded system is better than nothing, and safer than Kazaa. 

If Kazaa, Gnutella and others are severely damaged by the Crush the
Connectors attack, users will either give up free file-sharing, or
switch to less efficient social spaces. This might seem like an
unalloyed win for the RIAA, but for one inconvenient fact: there are
more people than are songs.

- There Are More People Than Songs

For the sake of round numbers, assume there are 500 million people
using the internet today, and that much of the world's demand for
popular music would be satisfied by the availability of something like
5 million individual songs (Apple's iTunes, by way of comparison, is a
twentieth of that size.) Because people outnumber songs, if every user
had one MP3 each, there would be a average of a hundred copies of
every song somewhere online. A more realistic accounting would assume
that at least 10% of the online population had at least 10 MP3 files
each, numbers that are both underestimates, given the popularity of
both ripping and sharing music.

Worse for the RIAA, the popularity of songs is wildly unequal. Some
songs -- The Real Slim Shady, Come Away With Me -- exist on millions
of hard drives around the world. As we've moved from more efficient
systems like Napster to less efficient ones like Kazaa, it has become
considerably harder to find bluegrass, folk, or madrigals, but not
that much harder to find songs by Britney, 50 Cent, or John Mayer. And
as with the shift from Napster to Kazaa, the shift from Kazaa to
socially-bounded systems will have the least significant effect on the
most popular music.

The worst news of all, though, is that songs are not randomly
distributed. Instead, user clusters are a good predictor of shared
taste. Make two lists, one of your favorite people and another of your
favorite songs. What percentage of those songs could you copy from
those people?

Both of those lists are probably in the dozens at most, and if music
were randomly distributed, getting even a few of your favorite songs
from your nearest and dearest would be a rare occurrence. As it is,
though, you could probably get a significant percentage of your
favorite songs from your favorite people. Systems that rely on small
groups of users known to one another, trading files among themselves,
will be less efficient than Kazaa or Napster, but far more efficient
than a random distribution of music would suggest.

- What Happens Next?

Small amounts of social file-sharing, by sending files as email
attachments or uploading them to personal web servers, have always
co-existed with the purpose-built file-sharing networks, but the two
patterns may fuse as a result of the Crush the Connectors strategy. If
that transition happens on a large scale, what might the future look
like?

Most file-sharing would go on in groups from a half dozen to a few
dozen -- small enough that every member can know every other member by
reputation. Most file-sharing would take place in the sorts of
encrypted workspaces designed for business but adapted for this sort
of social activity. Some users would be members of more than one
space, thus linking several cells of users. The system would be far
less densely interconnected than Kazaa or Gnutella are today, but
would be more tightly connected than a simple set of social cells
operating in isolation.

It's not clear whether this would be good news or bad news for the
RIAA. There are obviously several reasons to think it might be bad
news: file-sharing would take place in spaces that would be much
harder to inspect or penetrate; the lowered efficiency would also mean
fewer high-yield targets for legal action; and the use of tools by
groups that knew one another might make prosecution more difficult,
because copyright law has often indemnified some types of
non-commercial sharing among friends (e.g. the Audio Home Recording
Act of 1992).

There is also good news that could come from such social sharing
systems, however. Reduced efficiency might send many users into online
stores, and users seeking the hot new song might be willing to buy
them online rather than wait for the files to arrive through social
diffusion, which would effectively turn at least some of these groups
into buyers clubs.

The RIAA's reaction to such social sharing will be unpredictable.
They have little incentive to seek solutions that don't try to make
digital files behave like physical objects. They may therefore reason
that they have little to lose by attacking social sharing systems with
a vengeance.  Whatever their reaction, however, it is clear that the
current environment favors the development and adoption of social and
collaborative tools, which will go on to have effects well outside the
domain of file-sharing, because once a tool is adopted for one
purpose, it often takes on a life of its own, as its users press such
social toosl to new uses.

-=-

* Worth Reading =======================================================

Townsend on telecommunications and cities

  Anthony Townsend, (un)wirer of NYC parks, has published his doctoral
  thesis on the relationship between urban areas and telecom. It's
  interesting reading for anyone who cares about telecommunications
  regulation, or is interested in the effect of WiFi on cities.

  Townsend covers a lot of background, including the delightful
  insight that telephones made skyscrapers possible, because physical
  messaging between floors would never have scaled. He also notes that
  though telecom regulation is typically taken at the state and
  national level, it is really the municipalities where the critical
  choices take place.

  The background is mostly there to set the stage for Chapter 6, which
  outlines the potential of Wifi to alter urban space yet again, this
  time by further speeding up the "urban metabolism":

    In cities, it was also clear that the urban environment generated
    an enormous amount of information that needed to be anticipated,
    reacted to, and incorporated into everyday decision-making.
    Information about constantly changing traffic, weather, and
    economic conditions could be better transmitted through mobile
    phones and other wireless digital media. Traditionally, cities had
    functioned on a daily cycle of information flow with mass media
    like newspapers, third spaces like bars and cafes, and family
    conversations at the dinner table as the primary means of
    information exchange. With ubiquitous untethered communications,
    however, this old cycle was dramatically speeded up. As the
    information cycle sped up, there was a corresponding increase in
    the rate of urban metabolism -- the pace at which urban economic
    and social life consumed information and materiel. In effect,
    instead of the synchronous daily rhythm of the industrial city
    coordinated by standardized time, untethered communications were
    leading to a city coordinated on the fly in real-time.

  Worth a read. http://www.mit.edu/~amt/

* End ====================================================================

This work is licensed under the Creative Commons Attribution License.
The licensor permits others to copy, distribute, display, and perform
the work.  In return, licensees must give the original author credit.

To view a copy of this license, visit 
http://creativecommons.org/licenses/by/1.0 

or send a letter to
Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

2003, Clay Shirky