Azure Storage Studying Notes - NETSEC

Latest

Learning, Sharing, Creating

Cybersecurity Memo

Saturday, October 30, 2021

Azure Storage Studying Notes

Azure Storage is a Microsoft-managed cloud service that provides storage that is highly available, secure, durable, scalable and redundant. Whether it is images, audio, video, logs, configuration files, or sensor data from an IoT array, data needs to be stored in a way that can be easily accessible for analysis purposes, and Azure Storage provides options for each one of these possible use cases.



Within Azure there are two types of storage accounts, four types of storage, four levels of data redundancy and three tiers for storing files.

Two types of storage accounts:

  • standard
  • premium
Four types of storage:

  • Blob storage
  • File storage
  • Table storage
  • Queue storage

Four levels of data redundancy

  • LRS - Locally-redundant storage
  • ZRS - Zone-redundant storage
  • GRS/RA-GRS - Geo-redundant storage (GRS) 
  • GZRS/RA-GZRS - Geo-zone-redundant storage (GZRS)

Three tiers:

  • Hot
  • Cool
  • Archive


Storage Accounts

An Azure storage account contains all of your Azure Storage data objects: blobs, file shares, queues, tables, and disks. The storage account provides a unique namespace for your Azure Storage data that's accessible from anywhere in the world over HTTP or HTTPS. Data in your storage account is durable and highly available, secure, and massively scalable.

Azure Document : How to create a storage account.

A storage account provides a unique namespace in Azure for your data. Every object that you store in Azure Storage has an address that includes your unique account name. The combination of the account name and the Azure Storage service endpoint forms the endpoints for your storage account.

Azure Storage Account

The Azure Storage platform comprises the following data services:

  • Azure Blobs are an immensely scalable object store for text and binary data. Also includes support for big data analytics through Data Lake Storage Gen2.
  • Azure Files are organized file shares for cloud or on-premises deployments.
  • Azure Queue is a messaging store for consistent messaging between application components.
  • Azure Tables are NoSQL store for schema-less storage of structured data.
  • Azure Disks are block-level storage volumes for Azure Virtual Machines.

Type of Storage Accounts:
  • Standard general-purpose v2
  • Premium block blobs
  • Premium file shares
  • Premium page blobs


Note: https://cloud.netapp.com/blog/azure-anf-blg-azure-storage-limits-at-a-glance

Resource

Regions

Limit per Storage Account

Number of storage accounts per region

N/A

250

Maximum storage capacity

All

5 PB

Maximum request rate

All

20,000 requests per second

Maximum ingress

US and Europe

10 Gbps

Maximum ingress

Other Regions

5 Gbps**

Maximum egress

US

50 Gbps*

Maximum egress

Other Regions

10 Gbps**

Maximum number of virtual network rules

All

200

Maximum number of IP address rules

All



Container

A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs.


Azure Blob Storage

Blob storage includes three types of resources explained below:

Diagram showing the relationship between a storage account, containers, and blobs

The following points describe the use case scenarios:

  • Serving images or documents directly to a browser
  • Storing Files for distributed access
  • Streaming video and audio
  • Writing to log Files
  • Storing data for backup, restore, disaster recovery and archiving
  • Storing data for analysis by an on-premises or Azure-hosted service

Azure storage offers various access tiers, which allows storing the blob object data in a very cost-effective manner. The available access tiers include:

  • Hot– Augmented for storing frequently accessed data.
  • Cool– Optimized for storing less frequently accessed data, and the storage period lasts for at least 30 days.
  • Archive– Enhanced for storing rarely accessed data and the storage period lasts for at least 180 days with flexible latency requirements


Speed Performance:

A single blob supports up to 500 requests per second. If you have multiple clients that need to read the same blob and you might exceed this limit, then consider using a block blob storage account. A block blob storage account provides a higher request rate, or I/O operations per second (IOPS).

You can also use a content delivery network (CDN) such as Azure CDN to distribute operations on the blob. For more information about Azure CDN, see Azure CDN overview.



Limits for Blob storage:

Resource

Type of Blob

Limit

Maximum size of single blob container

 

5 PB

Maximum number of blocks

Block / Append

50,000

Maximum block size

Block

100 MB

Maximum total block size

Append

4 MiB

Maximum total blob size

Block

Approx. 4.75 TB

Maximum total blob size

Append

Approximately 195 GB

Maximum total blob size

Page

8 TB

Maximum stored access policies per container

All

5

Target request rate for blob

All

500/second

Target throughput for blob

Page

60 MB/s

Target throughput for blob

Block

Depends on storage account ingress/egress limits




Azure Files Storage

Azure Files provide fully managed File shares in the cloud that are approachable via the industry-standard SMB. Azure File shares can be attached parallelly by cloud or on-premises deployments of Windows, Linux, and macOS. It can be cached on Windows servers with Azure File Sync for quicker access. It permits the user to set up highly obtainable network file shares that can be accessed by using the standard Server Message Block (SMB) protocol. Multiple VMs can share similar files with both read and write permissions.

image alt text

The only contrast between Azure Files and files on a corporate file share is, user can access the files from anywhere by using a URL that points to the file and contains a shared access signature (SAS) token. SAS tokens can be generated by the user; they allow specified access to a private asset for a specific time period.

Azure Storage Files

Storage tiers

Azure Files offers four different tiers of storage, premium, transaction optimized, hot, and cool to allow you to tailor your shares to the performance and price requirements of your scenario:

  • Premium: Premium file shares are backed by solid-state drives (SSDs) and provide consistent high performance and low latency, within single-digit milliseconds for most IO operations, for IO-intensive workloads. Premium file shares are suitable for a wide variety of workloads like databases, web site hosting, and development environments. Premium file shares can be used with both Server Message Block (SMB) and Network File System (NFS) protocols.
  • Transaction optimized: Transaction optimized file shares enable transaction heavy workloads that don't need the latency offered by premium file shares. Transaction optimized file shares are offered on the standard storage hardware backed by hard disk drives (HDDs). Transaction optimized has historically been called "standard", however this refers to the storage media type rather than the tier itself (the hot and cool are also "standard" tiers, because they are on standard storage hardware).
  • Hot: Hot file shares offer storage optimized for general purpose file sharing scenarios such as team shares. Hot file shares are offered on the standard storage hardware backed by HDDs.
  • Cool: Cool file shares offer cost-efficient storage optimized for online archive storage scenarios. Cool file shares are offered on the standard storage hardware backed by HDDs.


Limitations related to file shares

Resource

Standard file shares

Premium file shares

Minimum share size

N/A

100 GB

Maximum share size

5 TB by default, can be increased up to 100TB

100 TB

Maximum file size

1 TB

4 TB

Maximum IOPS

1,000 IOPS* - Can be increased to 10000 IOPS

100,000 IOPS

Maximum stored access policies**

5

5

Target throughput**

60 MB/sec** - can be increased to 300MB/S

Ingress 4,136 MB/s

Egress 6,204 MB/s

Maximum number of share snapshots

200

200

Maximum directory/file name length (chars)

2,048

2,048

Maximum hard links

N/A

178


Azure Blob Storage vs File Storage

Azure Blob Storage and File Storage, both services have their own defined properties and are implemented in different scenarios. Azure Files provides fully managed and organized cloud file shares that can be accessed from anywhere. Azure Blob Storage permits the storage of unstructured data and it can be accessed at a massive scale

Consider a development environment where every developer needs access to IDE and tools without using the internet to download it. In this situation, Azure Blob Storage would meet the need and using which the developer can only store development tools then give a link to the team to access the Blob location.

For implementing a File server in an organization, the user should choose the Azure Files option. A File server is used to share Files across departments in an organization. When it comes to File sharing, end user should not be allowed to access the copies of the file from its URI and need to be mapped locally in the computers. This is when Azure File Storage fits the organization’s need.



Azure Storage Queues

Azure Queue storage is an Azure service that implements cloud-based queues. Each queue maintains an inventory of messages. Application components access a queue employing a REST API or an Azure-supplied client library. Typically, you’ll have one or more sender components and one or more receiver components. Sender components add messages to the queue. Messages are retrieved from the front of the queue for processing by receiver components. The subsequent illustration shows multiple sender applications adding messages to the Azure Queue and one receiver application retrieving the messages. Storage Queues are part of the Azure Storage infrastructure, feature a simple REST-based GET/PUT/PEEK interface, providing reliable, persistent messaging within and between services.

Azure Storage Queues

Concepts of Queue service

Concepts of Queue service


User Case:
Imagine the user works as a developer for a major news organization that reports breaking news alerts. The company employs a worldwide network of journalists that are constantly sending updates through a web portal and a mobile app. A middle tier web service layer then takes those alert updates and publishes them online through several channels. However, it’s been noticed the system is missing alerts when globally significant events occur.

The middle tier provides plenty of capacity to handle normal loads. However, a look at the server logs revealed the system was overloaded when several journalists tried to upload larger breaking stories at the same time. Some writers complained the portal became unresponsive, and others said they lost their stories altogether. The user has spotted a direct correlation between the reported issues and the spike in demand on the middle tier servers.

Clearly, user needs a way to handle these unexpected peaks. At such a situation user doesn’t want to add more instances of the website and middle tier web service because they are expensive and, under normal conditions, redundant. They could dynamically spin up instances, but this takes time and they would have the issue waiting for new servers to come online.

This problem can be solved by using Azure Queue storage. A storage queue is a high-performance message buffer that can act as a broker between the front-end components and the middle tier. The front-end components place a message for each new alert into a queue. The middle tier then retrieves these messages one at a time from the queue for processing. At times of high demand, the queue may grow in length, but no stories will be lost, and the application will remain responsive. When demand drops back to normal levels, the web service will catch up by working through the queue backlog.






Azure Table Storage

Azure Table storage behaves as a service that stores structured NoSQL data inside the cloud, producing an attribute store with a schema less design. Because Table storage is schema less, it is easy to adapt your data because the needs of your application evolve. Access to Table storage data is fast and cost-effective for several sorts of applications and is usually lower in cost than traditional SQL for similar volumes of knowledge.

Azure Table Storage
Azure Table Storage Use case

Table storage is employed to store flexible datasets like user data for web applications, address books, device information, or other metadata that the service requires. User can store any number of entities in a table, and a storage account may contain any number of tables, up to the capacity limit of the storage account Azure Table Storage Pricing. Azure tables are perfect for storing structured, non-relational data. Real time uses of Table storage include:

  • Storing datasets that do not require complex joins, foreign keys, or stored procedures and may be denormalized for fast access
  • Quickly querying data using a clustered index
  • Accessing data using the OData protocol and LINQ queries with WCF Data Service .NET Libraries


Azure Disk Storage

Azure managed disks are block-level storage parts that are managed by Azure and used with Azure Virtual Machines. Managed disks are similar a physical disk in an on-premises server but virtualized. In managed disks, all you must do is specify the disk size, type, and provision the disk. Once the disk is provisioned, Azure handles the rest. Each disk can take one of three roles in a virtual machine:

  • OS disk. One disk in each virtual machine contains the operating system files. When user creates a virtual machine, he/she selects a virtual machine image and that fixes the operating system and the OS disk that’s attached to the new machine. The OS disk has a maximum capacity of 2,048 GB.
  • Data disk. User can add one or more data virtual disks to each virtual machine to store data. For example, database files, website static content, or custom application code should be stored on data disks. The number of data disks that can be added depends on the virtual machine size. Each data disk has a maximum capacity of 32,767 GB.
  • Temporary disk. Each virtual machine contains a single temporary disk, which is used for short-term storage applications such as page files and swap files. The contents of temporary disks are lost during maintenance events, so do not use these disks for critical data. These disks are local to the server and are not stored in a storage account.
Azure Disk Storage Use case

The user manages a healthcare organization, and he is beginning a lift-and-shift migration to the cloud where many of their systems will be running on Azure virtual machines. These systems have a variety of usage and performance profiles which are highly confidential. The user is concerned about the storage and does not want to access that data outside the virtual machine.

To address these needs, the organization’s got to option is Azure Disk Storage. The Azure Disk Storage is capable of,

  • “Lift and shift” of applications that use native file system APIs to read and write data to persistent disks.
  • Preserve data that is not required to be accessed from outside the virtual machine to which the disk is attached.



References


No comments:

Post a Comment