Hi, I'm Kip Twitchell. Let me introduce myself first, as I begin this series of videos about my work life and the subject I've been interested in for now almost 30 years. The subject matter is not very interesting to most people deals with data, financial systems and automation. For some reason, I have found them interesting. I found them interesting now for a very long time. I'm going to try to explain what my insights have gained over that period of time.

Through this course of these videos. I'm going to try to make it a little more interesting than just a lecture. Recording in different places that I'm doing work that I'm doing as topics come up. I think I'll try to record those so that you get a little bit of flavor of the human interaction that comes with these very boring topics. Today, I'm in Hong Kong. I have a lovely view of Hong Kong harbor.

This first episode will be about unstructured and structured data. Okay, so let's begin. This simple analogy ignores some new data types like video and pictures and such. But if we talk about the vast majority of business data still today, textual, which is mostly unstructured and numeric, which tends to be structured, as simple example of textual data is the notes you're obviously taking on this video. Your text flows from one thought to the next, it might include paragraphs and bullets, but it might not. An example of numeric structural data is a list of transactions in a spreadsheet, like the voluntary contributions to me you're making for these awesome videos.

Usually, they will have column headings and values in each column are pretty well defined. Next, let's add another dimension to this simple division of data. Either textual or numeric data can be of high quality or low Quality, low quality might also simply be of questionable quality. Let's talk about textual data of questionable quality. First, it's not clear how accurate or truthful these narratives contents are. For example, if I make a blog posting on the internet about ancient Tibetan food rituals, that's probably not going to be worth very much.

I know nothing about that. Most of the information accessible in a search engine was given away for free or sold for advertising rates. Next, let's consider a high value textual quadrant. The highest value textual data is not given away for free or sold for advertising. Some of this information is confidential written in emails and memos. Some is sold, but not at advertising rates, things like subscribe periodicals or consulting reports.

I've written a couple of those in my time. I hope some of them are in this quadrant. IBM Watson is a supercomputer that combines artificial intelligence and sophisticated analytical software for very sophisticated question answering machine. When given access to high quality data, it excels at understanding unstructured textual data. When trained properly, it's primarily in this quadrant. Next, let's consider high quality numeric data quadrant.

We hope financial data qualifies as very high quality data. It moves trillions of dollars in the market every day, the basis of the workings of Wall Street. However, periodically financial reporting does not work correctly. When it falls into the lower question a quadrant, it can wreak havoc on the economics of those involved in that business. Let's focus on the high value quadrant for just a moment. Much more structured quantitative data was created from the 1950s through the 1980s.

Textual data started to take off with the creation of PC. Since then, perhaps More textual data has been created, there's certainly been a growth in structured data as well. new things have been automated for greater numbers of people and more and more commerce, that there's a difference between these two types of data. Although textual data has various versions and connections between it thoughts, words, meanings and those sorts of things. Structured Data has many more significant connections. Consider this simple example of a set of transactions with two different customers over a number of years.

Each of these transactions might have been independently independent of each other. yet very frequently, we need to know a balance of some kind, which is an accumulation of transactions. This connection between structured data is very, very common. You find it in your checking account or your credit card balance, your loan payments, even employee timesheet records and a host of other quantitative applications. Let's go back to the data volume again for a moment. Let's say the data represented in this table is the original transaction data.

If we add to it all of the balances needed to answer all types of question, which we're effectively duplicating the data, increasing significantly the amount of structured quantitative data in the world. The idea of business events can make clear basic principles about the world of quantitative data. Business events are more or less equal to transactions. We'll discuss business events in greater greater detail in subsequent episodes.