Relative scale — Big Data is a broad term for data whose size, arrival rate or forms strain the usual storage and analysis arrangements for the problem. Big is relative to resources and required response, not one universal number of rows. A small structured file on a capable machine need not require a distributed platform merely because its owner calls it big.
Volume and capacity — Volume concerns how much data must be retained or processed. If a dataset and required working state exceed one server's usable resources, partitioning across machines can become necessary. Disk capacity alone is not the only limit: memory, computation, access pattern, reliability and completion deadline influence a workable design.
Velocity and deadlines — Velocity concerns the arrival rate and timeliness requirements of changing or streaming data. A sensor feed needing a response within seconds differs from a historical archive processed overnight. A large stored archive is not necessarily high-velocity, and a smaller intense stream can be difficult despite modest retained volume.
Variety and representation — Variety includes different structures and media, such as tabular records, text, images, audio and events. Unstructured does not mean random or meaningless: useful structure may not already fit a fixed relational schema. Extracting entities, timestamps or features can create structured representations while retaining uncertainty about their interpretation.
Original sensor illustration — In an invented system, 1000 sensors each produce 20 records per second of 100 bytes each. Raw arrival is 2,000,000 bytes per second, or 2 MB/s using decimal units, before headers, indexes and replication. Over 60 seconds this is 120,000,000 bytes; the illustration is a calculation, not a claim about real school equipment or a universal Big Data threshold.
Sources and integration — Networked sensors, phones, service clicks and video can generate continuous data with different clocks, units, identifiers and quality. Integration needs explicit meanings and provenance. Combining a temperature number in Celsius with a Fahrenheit number without conversion creates a misleading apparent pattern even if both values are stored perfectly.
Quality before conclusions — Missing, duplicated, stale or systematically unrepresentative observations can distort analysis. More rows do not automatically remove bias or validate a proxy. Record source and relevant uncertainty, check impossible values and explain any exclusions; cleaning choices can themselves remove important cases if applied without regard to the purpose.
Relational fit and qualified limitations — A rigid row-and-column schema can be awkward for heterogeneous raw media or rapidly changing attributes, and distributing related transactions/joins across machines introduces complexity. This motivates alternative storage and processing models in the specification. It does not prove that all relational systems are incapable of distribution or that every large structured dataset should abandon SQL; choose according to the actual workload.
Pattern extraction and machine learning — Machine learning can identify patterns or useful features where manually specified rules or a simple tabular query are insufficient, for example grouping similar documents or recognising candidate objects in images. It is one important technique rather than a guarantee that every dataset needs a learned model. Training data, evaluation and relevant domain judgement determine what a result justifies.
Prediction, correlation and purpose — A correlation between features can support a useful prediction without showing that changing one causes the other. A model may exploit a spurious association or historical bias. State the intended task, assess results on appropriate separate evidence and avoid presenting a discovered cluster as proof of a natural or morally justified human category.
Distributed work, aggregation and reliability
Partitioning a workload — Partition data and work across machines so each handles an appropriate subset or operation. A coordinator can distribute tasks and combine results. A partition is not automatically a backup: if each record exists on only one failed machine, part of the data can become unavailable or lost despite having many other machines.
Parallel opportunities and dependencies — Independent per-record transformations can often run concurrently; work depending on previous global results needs coordination. Splitting a sequential dependency across ten servers does not automatically make it ten times faster. Communication, scheduling, imbalance and serial portions limit useful acceleration.
Map as transformation — A map applies a function to each input element and produces corresponding results. In a fictional click stream, map each valid event to (page,1), then combine counts by page. The function should define how invalid records are handled; silently ignoring all unfamiliar events can bias the interpretation of the resulting counts.
Grouping and reduction — Group related intermediate values by key, then reduce each group using an appropriate combining operation. For page events Home, Lab, Home, Quiz, Lab, Home, the counts are Home: 3, Lab: 2 and Quiz: 1. A global distinct-user total cannot generally be obtained by adding per-partition distinct counts when the same person appears in several partitions.Two disjoint event partitions produce partial page counts. A has Home: 2 / Lab: 1; B has Home: 1 / Lab: 1 / Quiz: 1; combining exact integer counts by page gives Home: 3 / Lab: 2 / Quiz: 1. Arrows show the explanation sequence, not a requirement that independent workers run one after another. Distinct-user counts would need identity-overlap handling.
Associative combining — An operation is associative when regrouping does not change its result: exact integer addition is a useful counting example. Distributed partial results can then be combined in different grouping structures when overflow is excluded. Subtraction is not associative: (10 − 3) − 2 = 5 while 10 − (3 − 2) = 9, so arbitrary regrouping changes its answer.
Order and numeric caveats — Associative does not mean every operation is commutative, and preserving relevant order can still matter. Finite-precision floating-point addition can give different rounded results under different groupings, so exact-integer reasoning does not automatically establish bit-identical floating results. Specify acceptable accuracy, numeric range and operation properties rather than promising universal reproducibility from parallelism alone.
Combining means correctly — For partition A with values 2 and 4, its mean is 3; partition B with value 12 has mean 12. Averaging the partition means gives 7.5, but the overall mean is (2 + 4 + 12)/3 = 6. Combine sufficient statistics, such as sum and count, to weight unequal partition sizes correctly.Partition A has 2, 4 with sum 6 / count 2 and mean 3; B has 12 with sum 12 / count 1 and mean 12. Averaging means equally gives 7.5, while combining sums/counts gives 18/3=6. The summary must retain the information required by the overall operation.
Replication and recovery — Replication stores appropriate copies to improve resilience, but it costs storage/communication and requires a consistency policy. A failed task can be retried if its inputs and required state remain available. Decide how duplicate execution affects external actions: repeating a pure calculation is different from charging an account twice.
Idempotence and retries — An operation is idempotent when repeating it has the same relevant effect as performing it once. Setting a status to a particular value can be idempotent under the stated conditions, while incrementing a total is not generally so. Retry handling needs deduplication or another suitable strategy where repeated external effects would be harmful; functional style alone does not guarantee exactly-once delivery.
Load, movement and cost — A highly uneven partition can dominate completion time while other workers finish early. Moving large intermediate data between machines can cost more than the computation saved. Evaluate bottlenecks, network demand, storage and energy for a defined workload rather than equating more machines with cheaper or faster completion in every case.
Functional features and distributed correctness
Immutability — An immutable value or structure cannot be changed in place after creation under its model. A transformation creates a new result rather than changing a shared original. This reduces some shared-mutation hazards, but implementations still manage memory and may share unchanged structure internally; immutability does not require copying every byte on every operation.
Pure functions — A pure function's result depends on its explicit inputs and it has no relevant observable side effects. Re-evaluating it with the same inputs gives the same result under the stated semantics. Reading a changing clock, mutable global variable or external service without representing that dependency in the inputs breaks this simple model.
Stateless transformations — A stateless worker does not rely on retained mutable history between independent requests for its answer. Passing needed state explicitly makes dependencies easier to see and tasks easier to distribute or retry. This does not mean the entire storage system has no state or that a database cannot be used anywhere in a functional application.
Avoiding shared mutation — Two workers changing one shared mutable total can overwrite or interleave changes incorrectly without appropriate control. Mapping inputs to independent outputs and combining defined partial results avoids that particular shared-update mechanism. Other errors remain possible, including incorrect functions, invalid inputs, communication failures and unsuitable aggregation.
Higher-order functions — A higher-order function takes a function as input or returns a function as output. A reusable map operation can accept different transformations while retaining its traversal structure. Passing a function is different from merely passing the numeric result of calling it once; this distinction helps express reusable parallel operations.
Referential transparency and tests — Referential transparency means an expression can be replaced by its value without changing relevant program behaviour. This makes local reasoning and deterministic tests easier. Test examples, boundaries and properties of pure transformations independently of task placement. Passing those tests does not prove the larger distributed application's security, fault recovery or data meaning.
Scheduling independence and limits — Pure independent tasks can run on different machines without depending on an accidental shared-update order. Their results still need correct combination, adequate resources and controlled external interactions. Functional programming supports distributed correctness and expression of parallel work; it is not proof that all algorithms become parallel or all runtime overhead disappears.
Fact-based information and graph structure
Single facts and identity — A fact-based model records individual assertions such as learner 101 hasName Ada or learner 101 booked workshop 201. Keep entity identity separate from a possibly changing/nonunique display name. One fact captures one piece of information under its representation, while provenance or time can qualify what the assertion actually means.
Original workshop graph — Represent learner 101 and learner 102 as Learner nodes and workshop 201 and workshop 202 as Workshop nodes. Directed BOOKED edges are 101→201,102→201 and 101→202; the last edge can carry bookingDate 2026-09-03. This is a fictional graph of associations, not proof that names themselves uniquely identify people or that an edge records every possible participation event.Original graph instance with learner 101 Ada, learner 102 Ben, workshop 201 Logic and workshop 202 Data. Directed BOOKED relationships are 101→201,102→201 and 101→202. Crossing drawn edges are not junctions or extra nodes. The 101→202 association can carry bookingDate 2026-09-03; display names are properties, not guaranteed unique identities.
Nodes and labels — A node represents an entity or relevant item; labels/types can describe roles such as Learner or Workshop. Properties attach values such as learnerID, name, title or capacity. A graph schema specifies allowed or intended structure and constraints, rather than being only a picture of the current sample data.
Edges and properties — An edge represents a relationship with relevant direction/type and may have properties, such as a date on BOOKED. Distinguish learner.name from BOOKED.bookingDate: the latter belongs to the association. An undirected friendship model and a directed booking model express different meanings; a line's direction must follow the declared relationship, not artistic preference.
Schema versus instance — A schema can describe Learner—BOOKED→Workshop and the relevant key/property expectations. A graph instance contains particular learner/workshop nodes and particular edges. A sparse sample with one booking does not prove a maximum of one; multiplicity and uniqueness require stated/enforced rules just as they do in relational modelling.
Traversal and query meaning — To find workshops booked by learner 101, follow that node's outgoing BOOKED edges to 201 and 202. To find learners sharing workshop 201, follow relevant incoming associations to 101 and 102, excluding self when the question requires other learners. A graph can have cycles or repeated routes, so traversal needs appropriate limits/visited handling rather than blindly recursing forever.
Parallel edges and history — If one learner can book, cancel and rebook the same workshop, a single unqualified edge may lose event history. Use suitable event identities, timestamps or related nodes/edge rules for the actual requirements. A property graph can allow multiple edges but this alone does not ensure duplicate events are intended or correctly distinguished.
Representation choices — The same simple booking facts can be represented through normalised relations or suitable graph structures. Graph representations can make relationship traversal natural, while relational schemas provide well-defined tabular operations and constraints. Neither representation proves all queries faster; workload, indexes, constraints, implementation and distributed placement matter.
Facts, claims and provenance — Calling a stored assertion a fact in the model does not prove it is true in the world. A sensor can be faulty and a person can enter the wrong name. Preserve source, date, uncertainty and correction rules where needed, and distinguish a direct observation, an inferred relationship and a disputed claim rather than flattening them into equal certainty.
Responsible extraction and evaluation — A larger interconnected dataset can reveal useful relationships while also increasing privacy and unfair-inference risks. Apply appropriate access, minimisation and evaluation in the intended context, and explain what the graph or model cannot establish. Match each chosen representation and processing method to a defined question rather than celebrating scale as an objective in itself.
Test yourself
50 questions · Random sets of 10. These quick checks support revision; practise longer explanations and justified judgements too.
Mr Kidman's Classroom
