Memory Hierarchy in Operating System

Memory Hierarchy in Operating System

7 mins read16.5K Views Comment
Updated on Sep 28, 2023 17:29 IST

Memory hierarchy helps in organizing the memory in a way that it can minimize the access time. It is based on a program behavior that is known as locality of references.


The memory hierarchy in OS is an essential concept in computer science. It helps to manage memory by organizing the various types of data into meaningful containers. The concepts behind a memory hierarchy can be difficult for newcomers because they involve complex algorithms and structures.

Table of Contents

Real-life Analogy

Suppose you have two baskets containing balls: A big basket and a small basket. You are supposed to find a particular ball. From which basket it will take less time? You can easily search for the ball from the small basket, which will take less time. So your preference will be to check the small basket first. It will take less time to find the ball, and if it is not found in the small basket, you will look in the big basket. And next what you can do is, that ball can be placed in the small basket for future use. The same concept applies to the memory hierarchy.

What is Memory Hierarchy?

A computer system is made of different parts. There is a CPU, several RAMs, hard disks, and other special devices. These parts are connected by buses and some other interconnecting devices so they can communicate with each other. 

Why Memory Hierarchy?

  1. Access time and cost balance

The main reason for using a memory hierarchy is to balance access time and cost. Less access time means more cost. Like registers are the smallest of all, their access time(time to fetch the data) will be faster, and their cost will be expensive. In the same way, secondary memory is the largest, so the access time will be more, but the cost will be less than other memories. In a nutshell, as the size increase, the access time also increases but the cost decreases.

2. Speed of communication

The CPU is responsible for fetching instructions, executing them, storing data, and controlling all other devices in the computer system. The speed at which the CPU can process data and execute instructions is much higher than the input rate of data from a hard disk. In other words, the CPU cannot keep pace with the incoming data. To solve this problem, computer system designers came up with a solution. They created a mechanism called a memory hierarchy that allows the CPU to get instructions and data from fast memory like registers and cache and process them at a higher speed so that the CPU can keep up with the data rate. A question arises here.

Can’t we use cache memory or RAM only?

The answer is NO, as cache and RAM don’t have much storage capacity. And the data we store is in terabytes. And having cache and RAM in terabytes will make your computer very costly.

Characteristics of the Memory Hierarchy

1. Capacity: 

This refers to the total amount of data that the memory can hold. The capacity rises from the top to the bottom of the hierarchy.

2. Access Period:

The time between a read-write request and a resource’s availability data. The access time decreases from top to bottom as the hierarchy rises.

3. Execution:

When the computer system was first created, Memory Hierarchy wasn’t used. This causes the system’s performance to decline. A system, therefore, improvement was necessary. This upgrade was made because of the Memory Hierarchy Design.

Threads and Multi-Threading: Operating system
Threads and Multi-Threading: Operating system
In any software system, there are always a variety of threads at work. These threads can be classified into two general types: user threads and kernel threads. User threads more
Kernel and its Types: Explanation with Real-life Analogy
Kernel and its Types: Explanation with Real-life Analogy
The Kernel is seen as the most important part of your operating system. It’s responsible for managing your system resources, running processes, and communicating with the hardware. The Kernel more
Working of Demand Paging in Operating System
Working of Demand Paging in Operating System
This article explain a very important topic of operating system-Demand paging with real-life analogy

Also explore: What is Operating Systems (OS) – Types, Functions, and Examples

Also explore: Operating system interview questions

Different levels of Memory Hierarchy

There are two types of hierarchy in computer systems i.e. data hierarchy and code hierarchy. Data hierarchy is nothing but the hierarchy of the data. Code hierarchy is the hierarchy of instructions. All computer systems have three levels of data and code hierarchies. They are –


Suppose the CPU wants to access data then it will try to find the data in registers. If not found, it will search in the cache memory; if not found there, it will move to search in the main memory and if not found, it will go to secondary memory in the worst case.


A register is a storage area on a computer that holds data temporarily. Registers are used to accelerate certain operations by holding the results of the operation in a local memory location. When the data is needed, the results are retrieved from the register and transferred to processor memory.

Cache Memory

Cache memory is used to store instructions and data that are frequently accessed by the CPU. It is faster than the main memory in computer. When you are operating on a computer, the computer first looks in the cache memory to see if the data is stored in the cache memory. If the data is found in the cache memory, it is copied to the CPU, and the CPU uses it. This avoids needing to go to the main memory and fetch the data. If the data is not found in the cache memory, it is fetched from the main memory and copied to the cache memory. If the CPU frequently accesses a particular set of instructions and data, then one part of the cache memory can be reserved for that data. So, the CPU always finds it in the reserved cache memory and does not have to fetch it from the main memory. This process is called cache optimization.

Main Memory

Main memory is a slower device than cache and registers in the computer system. It is the device where all the data is processed. But unfortunately, the speed at which it processes data is very slow. So, to make the system faster, designers created a memory hierarchy. One of the ways to make the system faster is to move the data from the main memory to a faster device. This faster device is called cache memory. So, the data is moved from the main memory to the cache memory so the system can process it faster.

Magnetic Disk


Magnetic disks store data on a spinning platter that a computer can read. They are the slowest devices in a computer system. They are used for storing data that is not frequently accessed. A magnetic disk is a device that stores data on a rotating medium such as a hard drive or an optical disk. The information is recorded on the disk as small magnetic fields.

Data is stored on a disk in sectors. A sector is the smallest allocation unit on a disk. 

A disk can be read or written to by using a drive. A read operation reads data from the disk and transfers it to the computer’s memory. A write operation writes data to the disk.

Magnetic Tape


Magnetic tapes are used to store data that is infrequently accessed. This data is usually archived data. It is also used to store data that is too large to be stored in the computer system. Magnetic tapes are similar to magnetic disks, but they use an electric current instead of a magnet to keep the data stored on them. 

All About Secondary Storage Devices
All About Secondary Storage Devices
Secondary storage devices are any non-volatile storage device that is either internally or externally present within the computer. These can be an7 device that helps provide permanent data storage.
Difference Between Primary Memory and Secondary Memory
Difference Between Primary Memory and Secondary Memory
Computer memory is simply the computer’s brain where data and information are stored for easy retrieval. Memory is the computer’s storage space that temporarily or permanently stores data or more
All About Primary Storage Devices
All About Primary Storage Devices
ROM and RAM are two main types of primary storages. ROM is non volatile that allows saving the data even if the computer is switched off. RAM is volatile more


Memory hierarchies are used in operating systems, database management systems (DBMS), web servers, search engines, and other applications that need to store or use large amounts of data. In this article, we discussed memory hierarchy with a real-life analogy. If you like it, then share it with your friends.

Happy Learning!!!


How does the memory hierarchy impact system performance?

The memory hierarchy is designed to reduce the average access time to data by placing frequently accessed data closer to the CPU. This helps in minimizing the time spent waiting for data to be fetched from slower levels of memory. By utilizing faster memory units for frequently accessed data, system performance and overall efficiency are improved.

Can the operating system directly control the memory hierarchy?

The operating system plays a crucial role in managing and coordinating the use of different levels of the memory hierarchy. It determines how memory is allocated, tracks the availability of memory resources, and uses caching algorithms to optimize data movement between different levels of the hierarchy.

Can the memory hierarchy be customized or extended?

The memory hierarchy is defined by the hardware architecture of a computer system. However, it can be customized or extended by adding additional levels of cache memory or using specialized memory technologies to improve performance in specific scenarios, such as in-memory databases or high-performance computing environments.

What is Virtual Memory?

Virtual memory, as commonly used, refers to paging. It is a mechanism where the OS provides an illusion to an application that it has a large amount of memory, but in reality, it's using a combination of RAM and disk space. When the RAM gets filled, the OS moves some of the data to the disk, making space in RAM for the application to run. The data moved to the disk is then fetched back into RAM when needed. This process is transparent to the application.

About the Author

This is a collection of insightful articles from domain experts in the fields of Cloud Computing, DevOps, AWS, Data Science, Machine Learning, AI, and Natural Language Processing. The range of topics caters to upski... Read Full Bio