The Now Economy

The following are only a few lines of excerpts from an extremely important argument about the “culture of design” surrounding software. It is a critical aspect of any effort to design “software that works the way society works,” to cite the credo of the decentralized software architecture crowd.

It may have an important impact on how CommerceNet Labs refines its own mission, too…

Software That Lasts 200 Years

In many human endeavors, we create infrastructure to support our lives which we then rely upon for a long period of time…

By contrast, software has historically been built assuming that it will be replaced in the near future (remember the Y2K problem). Most developers observe the constant upgrading and replacement of software written before them and follow in those footsteps with their creations…

In accounting, common depreciation terms for software are 3 to 5 years; 10 at most. Contrast this to residential rental property which is depreciated over 27.5 years and water mains and brick walls which are depreciated over 60 years or more… I can go to city hall and find out the details of ownership relating to my house going back to when it was built in the late 1800’s.

[Dan Bricklin] will call this software that forms a basis on which society and individuals build and run their lives “Societal Infrastructure Software”. This is the software that keeps our societal records, controls and monitors our physical infrastructure (from traffic lights to generating plants), and directly provides necessary non-physical aspects of society such as connectivity.

What is needed is some hybrid combination of custom and prepackaged development that better meets the requirements of societal infrastructure software.

How should such development look? What is the “ecosystem” of entities that are needed to support it? Here are some thoughts:

• Funding for initial development should come from the users…

• The projects need to be viewed as for more than one customer… Funding or cost-sharing “cooperatives” need to exist.

• The requirements for the project must be set by the users, not the developers. The long-term aspects of the life of the results must be very explicit…

• … Impediments such as intellectual property restrictions and “digital rights management” chokepoints must be avoided…

• The actual development may be done by business entities which are built around implementing such projects, and not around long-term upgrade revenue…

• The attributes of open source software need to be exploited. This includes the transparency of the source code and the availability for modification and customization… The availability of the source code, as well as the multi-customer targeting and other aspects, enables a market for the various services needed for support, maintenance, and training as well as connected and adjunct products.

• The development may be done in-house if that is appropriate, but in many cases there are legal advantages as well as structural for using independent entities..

• Unlike much of the discussion about open source, serendipitous volunteer labor must not be a major required element. A very purposeful ecosystem of workers, doing their normal scheduled work, needs to be established to ensure quality, compatibility, modifications, testing, security, etc… The health of the applications being performed by the software must not be dependent upon the hope that someone will be interested in it; like garbage collecting, sewer cleaning, and probate court judging, people must be paid.

The ecosystem of software development this envisions is different than that most common today. The details must be worked out. Certain entities that do not now exist need to be bootstrapped and perhaps subsidized. There must be a complete ecosystem, and as many aspects of a market economy as possible must be present.

Jeremy Zawodny sees a tipping point for feeds coming soon:

Real-time pings mean that we don’t have to wait for a full polling or crawling cycle before getting the latest content… Once this feed stuff hits the tipping point (I think we’re close), things will get really, really interesting. Suddenly these feed sources will be the thing people care about. The model of “search and find” or “browse and read” will turn into “search, find, and subscribe” for a growing segment of Internet users and it will really change how they deal with information on the web. What’s that gonna be like? Will the “web search” folks be ready? What about the browser folks?

The ability for a person or program to subscribe (and then get told when things happen, and take action as needed) is the foundation of The Now Economy. RSS and Atom can provide semi-structured data on which to take action — for example, for use in catablog-style commerce interactions.

Through John Battelle we discovered Rick Skrenta’s post on the subject of information feeds. Says Skrenta, “The proliferation of incremental content sources, all pumping out new material on a regular basis, is what the mainstream Internet user will consume.”

This in turn reminds us of Phil Windley’s recent observation that subscription-based information routing (such as that of mod-pubsub and KnowNow) allows applications to receive such streams of semi-structured information and then do something with them (such as filtering, aggregating, displaying, further routing, or taking action based on rules).

Such programming models will be ripe for exploration in the coming years as the applications of The Now Economy are discovered, developed, and deployed.

There are some great little infographics in the article quoted at length below. I found it in a binge of reading on Business Intelligence/Analytics, which yielded a few urls (del.icio.us anyone?

Great picture: http://www.insightful.com/products/iminer/Mortgage-screenshot_800pixe.gif

http://www.xmethods.net/ — you should visit in general if you haven’t already http://www.sas.com/apps/whitepapers/whitepaper.jsp — needs registration http://www.sas.com/solutions/sci/index.html — supply chain intelligence http://www.sas.com/industry/auto/warranty/index.html — failure correlation http://www.insightful.com/products/splus/unix.asp — the company that sells S+ now http://www.insightful.com/products/iminer/default.asp — cleanup and merge http://www.dmreview.com/portals/portal.cfm?topicId=230009 — analytics articles (also needs reg http://www.informationbuilders.com/ — customer highlights seemed interesting

Optimize Magazine > Gap Analysis > Real Time Means Real Change > August 2004

When asked which method has proven more effective in achieving real-time operation in their companies, only 16% of the 52 business and technology executives surveyed by Optimize Research cited investments in new, specialized technology solutions, such as grid or utility computing, which enable more distributed, flexible computing. By comparison, nearly half of the respondents said the more effective method has been to increase the efficiencies of existing IT solutions. Another 19% said both methods have proven equally effective, and 17% said neither has been effective. … But not many companies in the survey are doing much real-time monitoring or data collection. Executives were asked which processes or data types their company monitors in real time rather than by batch processes. The only response selected by a majority of the executives—58%—was Web site traffic/E-commerce activity. Fewer companies monitor real-time data on sales, customer interactions, inventory, customer shipments, output (products/services), and performance of individual software applications. Fewer than one-third collect information provided by business partners, data on incoming supplies, or product pricing information in real time. Only 19% of the executives rated their companies as extremely effective at monitoring real-time operations, while 75% said they were somewhat effective, and 6% not at all effective. Companies that are collecting and acting on data in real time are seeing benefits. Aerospace manufacturer Lockheed Martin Corp. in Bethesda, Md., had been relying on a mostly manual process of gathering data from multiple legacy applications for the procurement of materials for certain products, says CIO Joseph Cleveland. The cycle time for gathering data and making a procurement decision took weeks, and sometimes months, he says. To speed up the process, Lockheed Martin deployed EAI software to integrate the multiple systems used for materials management. Because workers can access and act on relevant data more quickly, the same process now takes only days.

In only a few weeks this blog has become the #1 entry in Google search for Now Economy.

On the other hand, we’re not even in the top 20 entries of Yahoo! search for Now Economy, though we are #2 in the Yahoo search for “Now Economy”, and we are #1 in the AlltheWeb search for “Now Economy”.

And, we’re not even in the top 20 entries of MSN search for Now Economy, though we are #7 in the MSN search for “Now Economy”.

The biggest hit for Now Economy that isn’t us is this Line56 piece by Max More on the Now Economy, followed by the earliest reference we can find to the Now Economy (namely, the GBN report from January 2001). I did also find this FoRK article by Rohit Khare in March 2002 about the “Now Economy”, as being pushed by McKinsey:

>it’s time to say hello to the “now economy.” Never heard of it?

You’re not alone. Even technology gurus sing different tunes when describing the newest buzzwords.

I will continue to egotistically arrogate the neologism “now economy” to myself and track its adoption

This is particularly cool to see in the McKinsey Quarterly. One of the class act aspects of the firm is that they track down even people who turn down a job offer and send out issues of the McK Q. Of course, now, it’s a lot cheaper favor that it’s electronic…

Internet News talks about Amazon’s forthcoming release of Amazon Web Services 4, pointing out the utility of shopping cart web services:

The Amazon shopping cart in AWS 4.0 now permits application users to add items to the Amazon Save for Later cart. Shopping cart abandonment continues to be a major problem for the e-commerce industry. A recent DoubleClick study showed that 57 percent of all carts are abandoned by shoppers and only 26.5 percent of them will come back to actually make a purchase.

What might come in Amazon Web Services 5? Mod-pubsub speculates “PubSub Amazon Web Services”, citing ZapThink senior analyst Jason Bloomberg:

Amazon Web Services 4 is still entirely request-reply in structure, which is adequate for supporting Web interfaces, but would not be sufficient or incorporating into more general business processes, or more broadly, into Service-Oriented Architectures, which require asynchronous Services.

Shopping Cart Services + PubSub Services = ?

Last night we stumbled on the Internet UPC Database, a hack that offers a public database of products and their Universal Product Codes. Anyone can submit new codes or search the database of codes. For example, here’s Diet Cherry Coke. Even more interesting is that a Google search for Diet Cherry Coke’s UPC number points you to the upcdatabase.com item. Which is even more astounding when you realize that the guy launched the service just seven months ago, and already has some extraordinary statistics:

Known Manufacturer Entries: 1058 UPC Entries: 805229 Distributable UPC Entries: 486416 (60.4%) Unique Mfr ID’s Represented: 44333 Average Items per Mfr ID: 18.2 Total size of database (approx.): 53.5 MB Update Requests Pending: 139

Too bad there’s no web service interface so we could start doing some interesting hacks…

The introduction to the paper below has a lengthy and illuminating rant about the pre-HIPAA patchwork of laws and regulations around medical records privacy. I’d definitely want to read the original NRC report…

HIMSS (Healthcare Information and Management Systems Society): JHIM: Journal of Healthcare Information Management

Security Measures Required for HIPAA Privacy

Margret Amatayakul, RHIA, FHIMSS

The state of security in healthcare is no less diverse. In 1997, the National Research Council released a landmark work: For the Record: Protecting Electronic Health Information. This report of a field study revealed that healthcare organizations did very little to counter security threats. Although it could not document the actual volume of threats, it did identify mistakes, improper use of access privileges, unauthorized use for spite or profit, unauthorized physical intrusion, and technical break-in as not uncommon occurrences. Likewise, organizational and even simple technical mechanisms such as authentication, auditing, access controls, and cryptography were rarely in place. Most healthcare organizations relied on corporate culture and closed networks to protect the private information of their patients and providers.

Some new attacks against the commonly-used SHA-1 and MD5 secure hash algorithms were announced at a rump session at the Crypto 2004 conference on Tuesday, as well as some less-commonly used secure hash algorithms, including the original SHA (now called SHA-0 to avoid confusion), RIPEMD, HAVAL-128, and MD4. Although these attacks in their present form probably won’t enable any system compromises, they have algorithm, protocol, and system designers looking around anxiously for alternatives.

As background, secure hash algorithms are intended to fulfill two requirements: first, that it be computationally infeasible to find two strings that hash to the same hash value (”collision-resistance”), and second, that it be computationally infeasible to find a string that hashes to a given hash value (”preimage-resistance”). Collision-resistance is clearly a stronger requirement, since you can construct a collision once you can find a preimage, but producing collisions does not necessarily imply that you can find a preimage for any given hash. Most systems don’t depend strongly on the collision-resistance property. Even without finding a flaw in the algorithm, there’s a property known as the birthday paradox that means that finding a collision by brute force takes a lot less work than finding a preimage by brute force.

None of the attacks provide a way to find a preimage for either SHA-1 or MD5, but collisions have been found for the first time in MD5, in a reduced-round version of SHA-1, and in SHA-0. I’m not sure whether collisions had previously been found in the other algorithms, but I don’t think so.

It was just becoming possible to find an MD5 collision by brute force, and there was a $10 000 bounty for a successful collision and a project to find a collision and claim the bounty through massively parallel distributed computing. The attacks presented at Crypto 2004 require substantially less computational work than the brute-force attack used in this project.

It was also strongly suspected that there were weaknesses in SHA-0, and some weaker attacks had been found on MD4 and MD5 in the past, making the breakage of those algorithms somewhat unsurprising.

So the only algorithm left standing is SHA-1, and even it looks weak, and there isn’t an obvious replacement, although Tiger, longer-hash versions of SHA-1, and AES-CBC have been suggested. Bruce Schneier has called upon the National Institute of Standards and Technology to initiate a search for a replacement algorithm, and

Hal Finney thinks

that at least the technique Joux used against SHA-0 could be used against a wide variety of secure hash functions.

The first version of the MD5 paper had some minor errors, which have been corrected in the current version (main page, PDF). Markku-Juhani O. Saarinen posted some thoughts and the extracted data files from the paper: 1, 2.

It’s interesting that none of the three papers presented at this conference were presented by a citizen of the United States; the MD5 (etc.) paper was published by a Chinese team headed by Xiaoyun Wang, the attack on SHA-0 was presented by French cryptographer Arnold Joux, and the attack on SHA-1 was presented by Israeli cryptographers Biham and Chen.

Cryptographic research in the US has suffered somewhat from the Digital Millennium Copyright Act. For example, Ian Goldberg, a prominent cryptographer who worked at the University of California at Berkeley before the Digital Millennium Copyright Act was enacted, explains why he left the United States in his comments on proposals to tighten Canadian copyright restrictions:

On an individual note, I have personally been involved in the mess that is the US DMCA; some of my own work as a cryptographic researcher, as well as that of my colleagues, has come under question as to whether merely publishing an academic paper is a violation of its anticircumvention provisions. Canada has developed a strong cryptographic industry, partially as a result of a more restrictive US legal regime in this area, and this industry, as well as our high quality of research and education, would be directly threatened if DMCA-like provisions were introduced here. I will not live or work in a country that imposes such restrictions on scientific inquiry. We must not allow academic speech to be chilled, stifled, and censored by any person, group of people, or industry.

Other cryptographers in the US have also scaled back their research to avoid running afoul of the DMCA. Who can say whether one of these attacks would have been discovered by an American cryptographer in the absence of that law? Our national security depends in part on our ability to deploy cryptographic algorithms, such as secure hash functions, before the intelligence services of other nations find a way to crack them. The DMCA may therefore be putting our national security in peril if, for example, Israel’s Mossad has progressed farther than Biham and Chen on cracking SHA-1.

(I took some other notes for this post.)

The Now Economy is a meme of many trends, not least of which is the import of mass customization and rapid prototyping’s role in the manufacturing cycle. In the middle of this Slate article is an excellent example of this vision:

Made to Order - How industrial design became a weekend hobby. By Clive Thompson

Do-it-yourself design will get really interesting when inventors are able to sketch something out and then hold the thing in their hands within a matter of minutes. Today, rapid-prototyping technology—that is, 3-D printers that can instantly crank out a physical copy of anything you design on a computer—is available only to elite design firms. It’ll get cheaper within years. Meanwhile, “original design manufacturing” companies overseas are becoming expert at quickly and cheaply cranking out MP3 players and laptops to specs set by brand-name firms like Virgin or Sony. Put those trends together, and it’s easy to envision an offshore service that will take my personal design for a music player and crank out 10 copies. Presto: the Clive brand MP3 player! Think of it as vanity electronics—casemodding on a superfast, global scale.

Dare Obasanjo:

A technology doesn’t have to solve every problem. Just enough problems to be useful. Two examples come to mind which hammered this home to me; Tim Berners-Lee’s World Wide Web and collaborative filtering which sites like Amazon use…

If you read the descriptions of the Xanadu model you’ll notice it has certain lofty goals. Some of these include the ability to create bi-directional links, links that do not break, and built-in version management. To me it doesn’t seem feasible to implement all these features without ending up building a closed system. It seems Tim Berners-Lee came to a similar conclusion and greatly simplified Ted Nelson’s dream thus making it feasible to implement and adopt on a global scale. Tim Berners-Lee’s Web punts on all the hard problems. How does the system ensure that documents once placed on the Web are always retrievable? It doesn’t. Instead you get 404 pages and broken links. How does the Web ensure that I can find all the pages that link to another page? It doesn’t. Does the Web enable me to view old versions of a Web page and compare revisions of it side by side? Nope.

Despite these limitations Tim Berners-Lee’s Web sparked a global information revolution. Even more interestingly over time various services have shown up online that have attempted to add the missing functionality of the Web such as The Internet Archive, Technorati and the Google Cache.

Here at CommerceNet we are grappling with the problem of solving just enough about decentralization to be useful in many commerce settings. More on that to follow in coming months…