NoSQL and Key-value databases
A recap from my last post: “Not Only SQL” databases, best known as NoSQL contains aggregate databases like key-value, document, and column friendly (Sadalage & Fowler, 2012). Aggregates are related sets of data that we would like to treat as a unit (MUSE, 2015c). Relationships between units/aggregates are captured in the relational mapping (Sadalage & Fowler, 2012). A key-value database maps aggregate data to a key, this data is embedded into a key-value.
Consider a bank account, my social security may be used as a key-value to bring up all my accounts: my checking, my 2 savings, and my mortgage loan. The aggregate is my account, but savings, checking, and a mortgage loan act differently and can exist on different databases and distributed across different physical locations.
These NoSQL databases can be schemaless databases, where data can be stored without any predefined schema. NoSQL is best for application-specific databases, not to substitute all relational databases (MUSE, 2015b). NoSQL databases can also have an implicit schema, where the data definition can be taken from a database from an application in order to place the data into the database.
MapReduce & Materialized views
According to Hortonworks (2013), MapReduce’s Process in a high level is: Input -> Map -> Shuffle and Sort -> Reduce -> Output.
Jobs: Mappers, create and process transactions on a data set filed away in a distributed system and places the wanted data on a map/aggregate with a certain key. Reducers will know what the key values are, and will take all the values stored in a similar map but in different nodes on a cluster (per the distributed system) from the mapper to reduce the amount of data that is relevant (MUSE, 2015a, Hortonworks, 2013). Reducers can work on different keys.
Benefit: MapReduce knows where the data is placed, thus it does the tasks/computations to the data (on which node in a distributed system in which the data is located at). Not using MapReduce, tasks/computations take place after moving data from one place to another, which can eat up the computational resources (Hortonworks, 2013). From this, we know that the data is stored in a cluster of multiple processors, and what MapReduce tries to do is map the data (generate new data sets and store them in a key-value database) and reduce (data from one or more maps is reduced to a smaller pair of key-values) the data (MUSE, 2015a).
Other advantages: Maps and reduce functions can work independently, while the grouper (groups key-values by key) and Master (divides the work amongst the nodes in a cluster) coordinates all the actions and can work really fast (Sathupadi, 2010). However, depending on the task division, the work of the mapping and reducing functions can vary greatly amongst the nodes in a cluster. Nothing has to happen in sequential order and a node can sometimes be a mapper and/or a grouper at any one time of the transaction request.
A great example of this a MapReduce Request is to look at all CTU graduate students and sum up their current outstanding school loans per degree level. Thus, the final output from our example would be Doctoral Students Current Outstanding School Loan Amount and Master Students Current Outstanding School Loan Amount. If I ran this in Hadoop, I could use 50 nodes to process this transaction request. The bad data that gets thrown out in the mapper phase would be the Undergraduate Students. Doctoral Students will get one key, and Master students would get another key, that is similar in all nodes, so that way the sum of all current outstanding school loan amounts get processed under the correct group.
Resources
- Hortonworks (2013). Introduction to MapReduce. Retrieved from https://www.youtube.com/watch?v=ht3dNvdNDzI
- My Unique Student Experience (2015a). Map Reduce: How it works. Retrieved from https://class.ctuonline.edu/_layouts/MUSEViewer/Asset.aspx?MID=1819533&aid=1819543
- My Unique Student Experience (2015b). The Pros and Cons of Schema-Less. Retrieved from: https://class.ctuonline.edu/_layouts/MUSEViewer/Asset.aspx?MID=1819533&aid=1819537
- My Unique Student Experience (2015c). Aggregate Data Models. Retrieved from: https://class.ctuonline.edu/_layouts/MUSEViewer/Asset.aspx?MID=1819533&aid=1819540
- Sadalage, P. J., Fowler, M. (2012-08-01). NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence, 1st Edition. [VitalSource Bookshelf Online]. Retrieved from https://bookshelf.vitalsource.com/#/books/9781323137376/
- Sathupadi, K. (2010) Map Reduce: A really simple introduction. Retrieved from http://ksat.me/map-reduce-a-really-simple-introduction-kloudo/
Dr. Skylar (Sky) Hernandez 