data collected from users provides added value.
And it’s not just companies using their own data, or the data contributed by their users. It’s increasingly common to mashup data from a number of sources.
When natural language processing fails, you can replace artificial intelligence with human intelligence. That’s where services like Amazon’s Mechanical Turk come in.
And it’s not just companies using their own data, or the data contributed by their users. It’s increasingly common to mashup data from a number of sources.
When natural language processing fails, you can replace artificial intelligence with human intelligence. That’s where services like Amazon’s Mechanical Turk come in.
“big data” is when the size of the data itself becomes part of the problem.
Information platforms are similar to traditional data warehouses, but different. They expose rich APIs, and are designed for exploring and understanding the data rather than for traditional analysis and reporting. They accept all data formats, including the most messy, and their schemas evolve as the understanding of the data changes.
Relational databases are designed for consistency, to support complex transactions that can easily be rolled back if any one of a complex set of operations fails. While rock-solid consistency is crucial to many applications, it’s not really necessary for the kind of analysis we’re discussing here.
are designed to be distributed across many nodes, to provide “eventual consistency” but not absolute consistency, and to have very flexible schema.
If anything can be called a one-stop information platform, Hadoop is it. Hadoop has been instrumental in enabling “agile” data analysis.
Edward Tufte’s Visual Display of Quantitative Information is the classic for data visualization, and a foundational text for anyone practicing data science.
visualization is key to data conditioning:
Visualization is also frequently the first step in analysis.
There are many packages for plotting and presenting data. GnuPlot is very effective; R incorporates a fairly comprehensive graphics package; Casey Reas’ and Ben Fry’s Processing is the state of the art, particularly if you need to create animations that show how things change over time.
They are inherently interdiscplinary. They can tackle all aspects of a problem, from initial data collection and data conditioning to drawing conclusions.
The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades.
“Information Platforms as Dataspaces,”
The Hadoop platform was designed to solve problems where you have a lot of data — perhaps a mixture of complex and structured data — and it doesn’t fit nicely into tables.
Hadoop is designed to run on a large number of machines that don’t share any memory or disks.
MapReduce: you map the operation out to all of those servers and then you reduce the results back into a single result set.
No comments:
Post a Comment