Archive | Data RSS feed for this section

… like a teenager with raging widget hormones

29 May

It has been hard to write of late. There is so much going on right now in terms of new social media experiences. My wife said it was so “cute” that I was just getting into Facebook now, after it had been open for 18 months already. I snapped back that it was really just opening up now. My spontaneous application promiscuity on its platform is embarrassing. I feel like a teenager with raging widget hormones.

It is a lot of work to express yourself uniquely online. You need to manage all of your various profiles across so many networks. Each network and engine represents a different source of traffic to your personal stream- Google, LinkedIn, WordPress, MySpace, Facebook, etc. Furthermore, they each provide different degrees of control over how your electronic likeness is distributed to others. In Media Futures speak, these are your various API’s (some which you control fully like your blog, others not at all like your PageRank) that together form your unique Algorithm identity in the online world.

This all makes sense, from the 30,000 feet perspective that I typically have written from in the intellectual capital that is New York City. But now that I meet with folks at Grove in the Marina instead of La Fortuna on West 71st street, I am both more engaged in the “real” world of Internet startups but also that much more conflicted by it.

While there are certain voices out here that call for radical transparency so as to keep any unsavory data mongers at bay, those same voices forever remain two cycles ahead of what gets funded and remain marginalized to watch as others commercialize their ideas from two cycles hence. The Internet is a data platform and therefore Internet businesses need to generate cash flow off of people’s data. This is the reality of the multi-billion dollar cookie cocoon that we are all clicking away within.

Still, regardless of how hopeful or hopeless the open Attention ecosystem proves to be, there are early traces of “mass market” Internet services paying Attention to Attention. For example, YouTube now enables me to “broadcast” what I am watching as a form of entertainment for others:

YouTube Active Sharing

And LinkedIn now enables me to see who else has visited my profile

Who has viewed my profile on LinkedIn

Granted these are small steps, but they are small steps by large players. Not to mention some truly open Attention thinking practiced by Google in both their Reader and Web History products that I will discuss in a coming post.

100 Billion Cookies and Nobody is Paying Attention

21 May

Despite all the hyperbole about the Internet advertising market M&A activities, I am surprised at the lack of critical perspective about the consolidation of cookies being placed and managed on users’ computers without their knowledge.

The recent spamacornucopia means more than $10 BILLION DOLLARS OF YOUR DATA IS BEING EXCHANGED AMONG BUYERS AND SELLERS THAT YOU DON’T CONTROL, starting with DoubleClick (and H&F their private equity owner) and Google, and then Right Media (Redpoint) and Yahoo!, and then 24/7 and WPP, and now aQuantive and Microsoft.

Cookie Wars- billions of dollars out of users control

I have heard that a profile is worth a dollar.

One could assume that a clickstream is worth $10.

We know, after all, that a mortgage lead can be worth more than $100.

How much is a cookie worth? As in, how much does it cost a company now on average to place a cookie on a user’s desktop? Of course the folks at Tacoda, Blue Lithium and Revenue Science would know this with more granularity, but my sense is that a cookie is currently worth about $.10. Please comment below if you have additional perspective.

And so, the $10 billion dollars worth of online advertising deals would equate to about 100 billion cookies served.

Do we as users have any sense of this reality, or any control over its consequences?

Wall Street 2.0?

24 Apr

 

Glocer in front of Media Futures

Tom Glocer,  CEO of Reuters, stands in front of  Media Futures at the Open Data Conference in NY

And so, what does exhibitionism have to do with Wall Street?

How does the voyueristic behavior of 20-somethings relate to the commission decisions of hedge fund masters of the universe?

Traditionally, very little.

Or at least we weren’t aware of these connections.  Now, however, the advent of personal surveillance technologies has begun to popularize processes that up until now have been unavailable to individuals.

This resonates with a comment that Reuters CEO Tom Glocer made at the Open Data Conference.  It was the night before the conference, over dinner, that Glocer gave his perspective on the evolution of "open data" in the context of financial services. 

He told a story about the transformation of individual data points into market data.  Surprisingly, he didn’t start with a traditional financial services firm, like Reuters, but rather with an individual Schwab customer.

This retail trader, by virtue of her decision as to what to buy or sell and at what price, is the most granular actor in the price discovery machine.  As Glocer told the story, the online retail investor was the proverbial butterfly flapping its wings in Hawaii causing hurricanes in China.  Her only action was to trade a stock in her 401K account online; but unbeknownst to her, Schwab took this trading data, along with that of all of the other individual retail investors, and established a higher level trend.  This process reverberated up through larger institutional brokers like Goldman Sachs and ultimately exchanges like the NYSE.   At each step up in aggregation and abstraction, significant economic value was extracted.  Although this individual’s behavior is too volatile in and of itself to offer much in the way of trend analysis, this does not mean that her behavior is worthless.

This is the foundation of Wall Street 2.0:  the individual data producer is beginning to wake up to the economic value she is creating.

This economic value had in the past been appropriated by those aggregating up the data from above.   Our electronic behavior, whether it be querying a search engine, clicking on an ad, checking out a stock, or trading a share, is generating value for other people that are in a position to aggregate and sell this information to institutions, who in turn transform it into some other form that ends up getting sold back to individuals.   Alchemy… to… Arbitrage.  This is nothing new.  What is new, however, is the extent to which our behavioral trails are no longer hidden, but are instead now available to us via various modes of personal Attention services, also known as myware.   This is the window that Open Data flows through:

Open data is to media what open source is to technology. Open data is an approach to content creation that explicitly recognizes the value of implicit user data. The internet is the first medium to give a voice to the attention that people pay to it. Successful open data companies listen for and amplify the rich data that their audiences produce.

Media Futures 2006: 3/5, API: Introduction

26 Oct

API stands for Application Programming Interface.  In the context of Media Futures, an API routes the output from one’s own unique Attention-processing Algorithm  into an Alchemical reaction triggered by the convergence of other human-driven APIs.

I wrote my first post about APIs in the Spring of 05, at a moment when APIs such as those of Flickr and del.icio.us were just starting to become becoming popular targets of developers.  Since then, the subject of APIs has become commonplace in any discussion of the future of media.  In fact AOL- that stalwart of old new media- is now obsessed with open APIs.  Tina calls it the “the liberation of egosystems.”  Open data transport has suddenly become de riguer among the even the most traditional media companies.  In less than two weeks, legions of their corporate development executives will descend upon SF to walk down the red carpet of the O’Reilly ceremony, ready to sign the top Web 2.0 talent to long-term studio deals.

But while we all fall over ourselves to proclaim our “openness,” we introduce a far heavier burden of trust into the mix.  Is one company’s “open” the same as another’s?  While I may be able to avoid data lock-in in that silo, how do i know for sure the next “open data” silo will be equally amenable to the mobility of my data?  These questions beg a deeper investigation into the history of APIs and their evolution both physically and electronically.

History

In a memorandum dated July 15, 1949, Warren Weaver, who held the position of director of the division of natural sciences at the Rockefeller Foundation from 1932 – 1955, wrote about the possibility of language translation by an electronic computer.  It was the first suggestion most had seen that such a thing might be possible, and as he draws the memorandum to a close, his words preview the emergence of the API:

Think, by analogy, of individuals living in a series of tall closed towers, all erected over a common foundation.  When they try to communicate with one another, they shout back and forth, each from his own closed tower.  It is difficult to make the sound penetrate even the nearest towers, and communication proceeds very poorly indeed.  But, when an individual goes down his tower, he finds himself in a great open basement, common to all the towers.  Here he establishes easy and useful communication with the persons who have also descended from their towers.

Thus may it be true that the way to translate from Chinese to Arabic, or from Russian to Portuguese, is not to attempt the direct route, shouting from tower to tower.  Perhaps the way is to descend, from each language, down to the common base of human communication – the real but as yet undiscovered universal language – and then re-emerge by whatever particular route is convenient.  Such a program involves a presumably tremendous amount of work in the logical structure of languages before one would be ready for any mechanization.

The key to examining the evolution of the role of the API in context of Media Futures lies, in fact, in the multiple resonances of its last term, Interface:  as a surface lying between two portions of matter or space, thus forming their common boundary; as a means or location of interaction between two systems or organizations; as an apparatus designed to connect two scientific instruments so that they can be operated jointly, the abstract concept of an interface contains in it the possibility of a very literal connection between two beings, two faces.  As a physical interface connects two pieces of hardware, a user interface connects a human and a computer and a software interface connects separate software components so that they may communicate with one another.  To interface is to come into interaction with a thing or being, to communicate, in manners both figurative and literal.

Mainframes

As a platform that allows a computer system, library or application to open itself to use by other computer programs, or to allow for the exchange of data between them, the APIs of yesterday were IBM mainframes and Microsoft SDKs, arcane languages of translation between hardware and software.

From the late 1950s through the 1970s, a number of American, German and British manufacturers (Burroughs, Control Data Corporation, General Electric, Honeywell, NCR, RCA and UNIVAC; Siemens and Telefunken; and ICL, respectively), produced such mainframes, computers used in large part by companies and government institutions for the purposes of bulk data processing in the context of, for example, the census or financial transaction processing.  IBM secured itself a position of power in the industry with the development of its 700/7000 series, based on vacuum tubes and transistors, and with its 360 series mainframe.  Unveiled in 1964, the 360 series was to be an all-around computer system, a series of compatible models for purposes both scientific and commercial – a series which, moreover, brought together features which were once only available in scientific or commercial computers, such as floating point arithmetic in the former and decimal arithmetic and byte addressing in the latter.  The 360 series also included supervisor and application mode programs and instructions and built-in memory protection facilities, making it one of the first computers manufactured with provisions specific to the use of an operating system.

Console of an IBM 360/67 mainframe

Mainframe

PCs

As the demand for the older mainframe systems fell off, new installations were seen mainly in the realms of finance and the government.  Personal computer networks came to challenge the mainframe.  It was during the rise of personal computing networks, though, that the APIs with which we are most familiar came into being and, in the case of Windows, achieved dominance. 

Altair

In 1975, the Altair 8800 was introduced in Popular Electronics, a personal computer that was affordable, user-friendly, and, some argue, the spark that set Apple Computer and Microsoft ablaze in their development of personal computers. The Apple II, though less capable and versatile than some of the larger computers of the day, gave computer enthusiasts an environment in which to develop their own programming skills and to operate simple office and productivity applications. 

Apple

The IBM PC released in 1981 took the personal computer into the realm of business, giving individual users word processing programs, spreadsheet programs and database programs which would change the way businesses stored, sorted and used their data. Four years later in 1985, in order to compete with the graphical user interfaces made popular by Apple, Microsoft released an add-on to MS-DOS – an operating environment known as Windows.

 

Windows

Though that release of Windows was not an operating system in the full sense of the term, it had pushed beyond the characteristics of a typical desktop, adopting some functions of operating systems.  Windows achieved a leg up on competing systems due in large part to the fact that MS-DOS dominated the early landscape of personal computing.  But the dominance of Windows (up until Google that is) is the API.  The APIs which enabled professional programmers to develop desktop applications on top of platforms (perhaps most notably the Microsoft Windows API), have now given way to APIs which feed off of the platform of the Internet.  And while Microsoft and the desktop are controlled by physical bodies, the Internet, despite the fact that certain companies do, in fact, oversee enormous pools of user data and have the ability to direct traffic as they see fit, is not governed by a particular body or set of bodies.  If the power flow of yesterday’s APIs was a vertical one, headed at top by the executives of companies like Microsoft, which allowed programmers to work off of their platform to develop applications to be used by the users at the  bottom, we might see the power flow of today’s APIs as closer to a horizontal one.   

Next:  The Thrilling Poverty of Physical Gestures

MEDIA FUTURES 2006: 2/5 ALGORITHM: History of Algorithm

3 Sep

An algorithm is a machine that can be used to reproduce a unique pattern of behavior.   The history of the word traces back to the Greeks and the instruments they used for mathematics; for example, the sieve.  In the context of Media Futures, imagine that algorithms are tightly woven filters that capture the full range of human Automata and slowly sift through them to produce the most meaningful, intentional gestures.

Animation_sieb_des_eratosthenes







Ancient Algorithms

Finding its root in algorism, a reading of the name of Abu Ja’far Muhammad ibn Musa Al-Khwarizmi, the 9th century Persian mathematician who described a set of rules for solving both Linear and Quadratic equations, algorithm came to its present state by way of an 18th century European Latin translation and soon expanded its meaning to encompass all definite procedures for solving problems or performing tasks.  The very first algorithms are a part of the Babylonian mathematical legacy – a legacy which not only left us with algorithms for factorization, finding square roots and performing long division, but which also left us with the base 60 system that gives 60 minutes to an hour, 60 seconds to a minute, 360 degrees to a circle and 24 hours to a clock.  Babylonians were in fact able to calculate things with the same accuracy as Renaissance mathematicians due to their use of number tables, like the Plimpton Tablet, a table of Pythagorean Triples from about 1700 B.C.

Plimptontablet

While the Babylonians based their mathematical system in large part on algebra, the Greek system of mathematics was heavily based upon geometry.  It is speculated, though, that the founder of Greek science and mathematics, the philosopher Thales of Milet, visited Egypt and Babylon during his lifetime (634 – 546 B.C.) and brought back knowledge of their astronomy and geometry.  The Egyptians made great contributions in the fields of medicine, astronomy and applied mathematics, and while the former triumphs are well documented, there exist no records of the process by which they reached their mathematical conclusions.  Thales built on the knowledge brought back from his trips, inventing deductive mathematics and proving a number of theorems – a circle is bisected by a diameter; the base angles of an isosceles triangle are equal; and pairs of vertical angles formed by two intersecting lines are equal.

The foremost text on geometry came from fellow Greek Euclid, whose Elements put together former geometric knowledge with definitions, postulates and opinions – and, of course, Euclid’s elegant and rigorous proofs of the above.  In that text, he discussed the algorithm for finding the greatest common divisor of two numbers, which is today referred to as the Euclidean algorithm.  One hundred years later around 200 B.C., the world saw the next great algorithm – the Sieve of Eratosthenes, which was used to find prime numbers.

Sieve

From Wikipedia: 

Sieve of Eratosthenes is a simple, ancient algorithm for finding all prime numbers up to a specified integer. It is the predecessor to the modern Sieve of Atkin, which is faster but more complex. It was created by Eratosthenes, an ancient Greek mathematician.

 

Another important site in the history of the algorithm was Alexandria, home to Hero, Ptolemy, and Diophantos.  Hero, whom we will remember as the inventor of the steam eolipile and other Automata, published widely on geometrics, optics and mechanics – as well as mathematics.  Though sources suggest his work is derivative of Archimedes and the work of the Babylonians, his Formula to calculate the area of a triangle in terms of its sides and his Method to extract a root are important contributions to the world of mathematics.  Ptolemy published widely on astronomy and geography and calculated the best approximation of ‘pi’ for his time.  And Diophantos, known as the ‘father of algebra’, wrote his thirteen-volume Arithmetica on the solution of algebraic equations and the theory of numbers and introduced the use of algebraic symbolism with an abbreviation for the unknown for which he was solving.

But Diophantos shares the title of the ‘father of algebra’ with the aforementioned Al-Khwarizmi, whose work was responsible for significant advances in the world of mathematics. 

Alkhwarizmi_kitab_large

It was Al-Khwarizmi’s work that promoted the use of Hindu-Arabic numerals that not only pushed forward the numeral system we use today, but that gives us the very term algorithm. From the very first algorithms of the Babylonians to those of Al-Khwarizmi – to John Napier’s 1614 method for performing calculations using logarithms to the 19th century work of Boole, Frege and Peano, which set out to reduce arithmetic to a series of symbols which could be manipulated by rules – to the work of Babbage, Lovelace and Turing, which took these rules and transformed them into agents of action in computing, these feats of problem-solving are instrumental in understanding man’s quest for a grasp of the workings of the world at large.      

Babbage and Turing

One great advantage which we may derive from machinery
is from the check which it affords against the inattention, the
idleness, or the dishonesty of human agents.
From Babbage’s 1832 work “On the Economy of Machinery and Manufactures”

In our discussion of rules that govern the Internet, we must turn to the work of Babbage and Turing, for it serves as the important foundation for computing at all.  Babbage’s work grew out in part out of a need for more accurate mathematical tables, which were essential calculating aids used in navigation and astronomy, insurance and civil engineering.  These tables were produced by human computers and by hand – and as such, they were prone to error in terms of computation and reporting.  Even the slightest errors in navigational or astronomical tables can be costly – so it is no surprise that in the years leading up to Babbage’s project, government sources were willing to fund projects that would minimize the costs of troubleshooting. 

For example, the British Nautical Almanac, the world’s first permanent table-making project – had a reputation for ever-improving accuracy since its inception in 1766.  But moving into the 19th century, that seaman’s bible swung into a dangerous territory of inaccuracy and error, and the British government recognized the promise of producing mathematical tables mechanically and typesetting them by the same machine. 

So Babbage set out, with financial support (and the admirals’ prayers) to improve the accuracy of those ever-important mathematical tables by constructing algorithm-driven machines.  It was a move that mechanized the production of thought, a move that would eliminate human folly in computation, transcription and typesetting.  The result would be better answers, answers which would in turn be used for giving new instructions, as inputs in other algorithms.   

Babbage never finished his Difference Engine – though, in 1832 his manufacturing engineer did construct a working portion of it, which measured two and a half feet high by two feet wide by two feet deep.  Babbage moved forward to conceptualizing what would be the world’s first programmable digital computer – the Analytical Engine.  Babbage’s designed the engine such that it would separate the sites of arithmetic computation from the storage of numbers.  The computation would be carried out through a series of steps recorded on punch cards, such as the ones used in the technology of the Jacquard loom. 

A_engine

But however intriguing and important the technology seemed, Babbage’s Analytical Engine – due to factors financial and logistical – was never built.  It comes to us only through Ada Lovelace’s annotated translation of a French introduction to the machine – a piece of writing that established the algorithm for the computation of Bernouilli numbers, and a piece of writing that established the idea of computer programming.  Turing would later build on the work of Lovelace and Babbage, formalizing their concepts in the Universal Machine.

When Turing introduced the mathematical description of the Universal Machine in the 1936 paper “On Computable Numbers”, he set out to answer the Entscheidungsproblem, the third question left by mathematician David Hilbert.  Gödel had already answered Hilbert’s first two questions – No, mathematics was not complete, and it was not consistent.  Turing showed that mathematics was not decidable.  And that recipe to solve a particular problem, gave us an answer that begs the asking of a new set of questions.

Follow

Get every new post delivered to your Inbox.

Join 14,441 other followers