Note Taking : Veeam v12 NAS Enhancement

 This is a blog post on note-taking for Veeam V12 NAS Enhancement

1. Backup to object storage directly

2. NAS Backup - Copy mode

• copy recent data to the archive repository

3. Immutability for NAS backup

4. Storage Integration - Nutanix Files support on Veeam

• NAS Filer
• Automated snapshot creation
• Changed File Tracking

5. NAS to tape - File to Tape with new improved engine

• NAS backup to tape does not support GFS
• Tape job always writes latest restore point to tape

6. Instant File Share Restore

1. SMB shares - no longer read only and can migrate to production
2. NFS shares - can be published as read only SMB

7. Health Check available for NAS

8. Rotated Drives - with limitation

• only for backup
• Not for archive repository, backup copy

Note Taking : Veeam v12 Tape Enhancement

This is a blog post taking for Veeam v12 Tape Enhancement

LTO 9 Support

Tape Server on Linux

• Not recommended running on hardened linux repository
• Put on a separate Linux machine
• Linux uses SCSI drivers used
• Tested on IBM
• Support x64 system only. Same as VBR repository/proxy in v12

New File to tape job

• Require to license - VUL
• 1 instance = 500GB
• Allow to backup directly to tape using file to tape job
• Better performance & optimization (parallel tape drive support, database optimization, backup and enumeration run at same time, asynchronous read)
• Retain permission like NAS Backup
• Cross ACL restore is not supported (Windows to Windows/Linux to Linux)
• Path exclusion supported
• Console display for file to tape job (file share). Able to restore file to tape job
• V12 last backup time is stored in UTC format

NAS Backup to Tape

• Avoid load on production NAS storage
• Fulfill 3-2-1
• Source -> NAS backup to Disk -> File to Tape
• NAS backup/and or NAS backup copy as source
• Tape job always writes latest restore point to tape
• NAS backup to tape does not support GFS
• A periodic full backup is available for file to tape/NAS to tape
• Files are store in native format. Restore files as files
• Does not consume license for NAS to tape

Other tape enhancement

• Support backup object storage to tape
• Daily GFS & Monthly media set
• Display current activity on tape drives
• Eject after inventory or catalog
• Audit for tapes
• Windows event
• File from tape restore audit
• General options > security > audit logs location
• Default to C:\ProgramData\Veeam\Backup\Audit

Note Taking: Veeam v12 Security Enhancement

This is a blog post on note-taking for Veeam V12 Security Enhancement

Your responsibility

1. Secure your infrastructure

2. Secure your data

3. Secure your session

4. Secure your application

5. Secure your visibility

Base security

• Support ipv6
• All supported except
• Veeam Backup for Nutanix and RHEV
• Plug in for AWS, Azure and GCP
• Unmanaged Veeam Agent
• Kasten - not tested

Data Security

• Any repository with immutability
• Hardened repository
• Object lock for object storage
• Storeonce catalyst
• Primary Backup and archive
• NAS backup immutability
• Enterprise Plug-in backup immutability

Authentication

• Group managed service account for Application-Aware Image processing
• Backup Server does not store password 
• Backup Server gets password on-demand from Active Directory
• Recovery Token for bare metal recovery on Veeam Agent

Application Security (Session)

• MFA for Veeam Console
• Auto Log off after X minutes

Visibility

• Classified data marking- by use tag/label. Required for security certification.
• New column in inventory: last backup. To identify who perform the action 

Security update subscription

• https://veeam.com/knowledge-base.html
• Select security advisory
• Enter your email address

Personal Experience Use AirAsia Ride Advanced Booking

I give AirAsia Ride a try since GrabCar does not allow for advanced booking. The only platform enabling you to do so at the time of writing this blog is 27 June 2022.

[Destination]

To Subang Airport to catch a flight to Penang.- Monday 27 June 2022

[Booking]

• Tried on Sunday (26 June 2022) morning, but few drivers accepted and then canceled my booking.
• Give up and on hold till night.
• At night- Tried booking. Managed to get a driver. No cancellation make.
• Booking with credit card payment

[Monday Trip]

• Able to see the driver on the way after booking time.
• Pick up by the driver

[Comunication with driver]

Based on his feedback:

AirAsia Ride - no penalty driver if cancel booking

If the driver cancels last minute, the system will try to find a nearby driver. If cannot find it, then no pick up. Therefore no guaranteed will get a ride

Driver prefer these payment methods

• Cash - most prefer
• Credit card - less prefer as payout needs to wait for a week
• AirAsia Pocket - not preferred due to already passed 3 weeks, yet to get money back from the previous ride.

Driver feedback on GrabCar:

• GrabCar has a penalty system to drivers. 
• No advanced booking at this moment.
• Can accept credit card as payout received by driver is 1 day

[Tip]

Encourage for the driver to select your ride

[After ride]

• Can rate the driver
• Get receipt by email

The strange part is driver added RM4 for toll even though they did not use toll. Cannot complain much about the system as traffic was bad in the morning to Subang Airport. Just consider as a tip to the driver.

Note Taking : Veeam v12 Core Architecture Improvement

 This blog post on v12 Core Architecture Improvement

Improvement

• Postgres Veeam Database
• Alternative from Microsoft SQL Express
• Move backup job from Veeam Console
• Execute active full/retry for one VM of a job
• Without executing a job for multiple VM
• Do Health Check outside the backup window
• Can set scheduling
• Improve per machine backup chain
• Clean up orphaned backup based on retention time. 
• Ipv6 support
• Prior version support ipv4
• Redundant Gateway
• Gateway for Dedup Appliance
• Multiple locations with different internet bandwidth
• Code name: VeeaMover - for migration storage to new hardware
• Simplified repository change -> Move Backup Button
• Moves backup files (including transaction log)
• Copy backup files
• Move machine between job
• SOBR Improvement
• Rebalance for all extents
• All extent in maintenance mode during rebalancing
• Export full backup from object storage
• Multiple Object Storage bucket in performance and Capacity Tier
• 50TB per bucket for most vendor
• Object storage of the same type (S3 != S2 compatible)
• Object storage cannot be mixed with non-object storage
• Amazon S3 or Azure Blob as Performance Tier
• optional capacity tier 
• Archive Tier: Amazon Glacier / Azure Archive
• More Better Performance for compression
• Default: Optional. Best setting- same no changes
• High compression - 3x faster backup, 20% better compression, restore 2x faster
• Exterme compression- 40% faster backup, 15% better compression, 2x faster restore
• New Roles and Features on Linux
• Linux Proxy - backup from storage snapshot for NFS
• CDP Proxy
• Tape Server
• Security
• Group Managed Service Account for Application-Aware Image Processing (GMSA support)
• Kerberos environment only
• Support AAIP Vmware VM, Hyper-V
• Windows Agent, Linux Agent
• Storage Plug in
• NFS 4.1 (repository and source share)
• SMB 3 (repository and source share)
• All veeam B & R components

Note Taking - Veeam v12 Veeam Agent Enhancement

 This is a blog post on note-taking for V12 for Veeam Agent. Total 5 Veeam Agent

1. Veeam Agent for Windows

• Object storage support directly. License required [paid version]
• Not needed synthetic full and compact full
• Installer size decreased from 364MB to 144MB in Beta2. Easier deployment with low bandwidth
• Efficient file-level backup for changed files. Block-level efficiency with file backup and only captured changed block of the changed file. Transfer only changed blocks.
• DB changed to SQLite - low resource consumption. Previously use SQL LocalDB

2. Veeam Agent for Linux

• Direct backup to object storage
• GFS support
• Postgres log backup support - only on Linux
• Snapshot based backup with LVM snapshot - Good for distribution not supported by veeamsnap
• Non SSH mode for Linux workload. No sudo/root credential on VBR, no need for SSH on Linux, No need for long sudoers list
• Veeam Explorer for Postgres & Enterprise Manager support

3. Veeam Agent for Mac

• New UI
• Backup directly to object storage
• Backup to different location with multiple jobs
• resume backup job

4. Veeam Agent for Solaris

• Bare metal recovery

5. Veeam Agent for AIX

• Bare metal recovery

6. Cloud Native Agent for AWS & Azure  [New] 

• For AWS and Azure Platform
• Application-Aware processing (eg database log shipping)
• Simplify networking (no VPN/direct connection VM needed)
• Suitable for customers that lacks permission to run Veeam Backup for Amazon/Azure
• Automatic discovery via cloud provider API
• Can use machine selection/Instance ID / tag based
• Distribution repository: upload agent setup components once, registered object storage repository in AWS/Azure.
• Not supported Windows Failover Cluster
• Managed by backup server

General

• Secure Bare metal recovery with the recovery token
• for security 
• valid for 24 hours per default. Will expire automatically

Note Taking : Veeam Backup for Microsoft 365 Best Practice

This blog post is my note-taking on Veeam Backup for Microsoft 365  Best Practice.

Security & Hardening

Patching 

Latest Windows Operating system patching on all veeam components

Authentication

1. Use Modern app-only authentication

2. Modern authentication with legacy protocols allowed [limited]

Veeam components

1. Workgroup / Join to domain for all veeam components

2. Self-signed certificate might not be allowed by enterprise customers. Prefer PKI (Internal PKI or External (Public) Certificate

Data Separation

1. Backup Copy 

Can put on different cloud provider

Encryption

Additional security with at rest AES 256 encryption for Microsoft 365 data in object storage

Note: Password loss protection is NOT available

Self Service Restore Portal

1. Dedicate administrator to restore administrator 

2. Access Self-Service Restore Portal for recovery. Avoid login to VM

Note Taking: Veeam Backup for Azure Best Practice

 This is my personal blog note-taking for Veeam Backup for Azure Best Practice

Backup Appliance

running Ubuntu

1. B2s (default) (2 vcpu, 4GiB RAM with 32GB Premium SSD data disk) -support 400 workload, 50 workers & up to 20 000 restore point.

2. F8s (medium) (8 vcpu, 16 GB RAM with 64GB premium SSD data disk) - support 1500 workload, 250 workers and 70000 restore point.

3. F16s (large) (16 vcpu, 32 GB RAM with 128 GB premium SSD disk) - support 1500 workload, 500 workers and 70000 restore point.

Maximizing throughput

Workload per policy - 50

Worker per policy and per storage account

Worker speed (F2s_v2 = ~100 Mib/s)

Azure storage account limit (10-60 Gbps depending on the account)

Azure API limits (1200 writes, 12000 reads of ARM API)

Worker per appliance (500 recommended, max 1000 per region)

Memory consumption per policy - 100MiB + 3 MiB per workload in policy: 250 Mib

Repository

Average size of backup data in object storage - 40% - 50%

Object size :

1.Backup data (hot & cool tiers) - 1 MiB compressed (~512 KiB)

2.Backup data (archive tier) - 512 MiB

3. Metadata - 4KiB per GiB of VM source data

Storage account limit on IOPS. Configure one backup repository per storage account

Workers

Leverage on ubuntu image. Deployed in the same resource group and subscription as Veeam Backup for Azure.

1. Creating backup/archive of Azure VM - region with target repository

2. Creating backup/archive for Azure SQL - region with Azure SQL database to be processed

3. Azure VM restore, SQL Restore, Volume level restore - region where restored data will reside

4. File level restore from snapshot - region where snapshot resides

5. File level restore from backup - region where backup repository resides

Worker size

Change the size of the data disk allocated to the worker instance: /etc/veeam/azurebackup/Config.ini

[WorkerVMDeploymentOptions]DataDiskSizeinGB =32 (default)

Recommended worker maximum based on testing

1. Recommended worker for default appliance size - 50

2. Maximum worker per region per appliance - 1000

3. Worker per service bus (two queue per worker, based on default basic tier) - 5000

4. Azure ARM API reads (per tenant/user/hour)* - 12000

5. Azure ARM API (per tenant/user/hour) * - 1200

* Azure Management APU request limit and throttling

Policies

1.Repository per policy -1

2.Worker per repository - 50

3.Appliance memory consumption ~50% RAM + memory used by policies

4.Policy memory consumption - 100 MiB per policy + 3 MiB per workload in

Calculate retention point

-> snapshot per workload for the first day

-> daily backup

-> weekly backup

-> monthly backup

-> yearly backup

Get total restore point per workload X Number of VM, to get a total restore point

Note Taking - Veeam Backup for AWS Best Practice

 My personal blog note about Veeam Backup for AWS Best Practice [24 June 2022]

Backup Appliance Size

1. T3.medium (default 2vcpu, 4 Gib RAM) - support workload 500 - 1000 , 50 Instance per policy

2. T3.2xlarge (medium - 8 vcpu, 32 Gib RAM) - support workload 1000 - 3000. Around 50 - 150 instance per policy

3. C5.9xlarge (large - 36 vcpu, 72 Gib RAM) - support workload 3000 - 4500. Around 50 - 150 instance per policy

Repositories

1. Use a dedicated IAM role (repository role)

2. Support for encryption via password or KMS Integration

3. KMS usage is advised. Password can get lost or forgotten (not recoverable)

Object storage data size

1. Average size of backup data in object storage - 40% - 50%

2. Backup data (S3 tiers) - 1 MiB compressed (~512KiB)

3. Backup data (Glacier tier) - 512 MiB

4. Metadata - 4KiB per GiB of VM source data

Workers

Deployed within the backup account

Worker provision is based on available vcpu count (AWS service quota/per region) 

On average between 10 to 40 workers per region

Different sizes are used for cost-effective protection

Worker is leverage on Ubuntu Image

Placement of worker

1. Creating backup/archive of instance - worker placement at region with target repository

2. Instance Restore/ Volume Level Restore - worker placement at region where restored data will reside

3. File Level Restore from snapshot - worker placement at region where snapshot resides

4. File Level Restore from backup - worker placement at region where backup repository resides

Policy Designing

1. Create specific IAM roles if possible per service

2. Use tag where possible

3. Consider properly your source and target for cost effective design

Policy 

1. 50 -150 workload per policy

2. Appliance memory consumption - 1.5 Gib and 5% RAM free + memory used by policies

3. Policy memory consumption - 100 MiB per policy + 3Mib per workload added

Every policy uses around 225MiB of RAM upon run (even with just 1 instance)

Formula:

Appliance RAM in MB * 0.95 - 1536 MiB - (225 MiB * N of policies + 3MB * N of instances in the policy)

4. Don't start all jobs concurrently unless got sufficient resources.

Security

1. Use cross-account/region - isolate backup

2. Integrate with IAM roles (dedicated)

3. Enable Encryption to  safeguard against internal & external threats

4. Use Amazon KMS to easily control secure access to encrypted backup data

5. Use RBAC to delegate permission to administrate and perform tasks

6. Use MFA to protect access using a second source of validation

Video: Restore M365 from Archive Storage Using Veeam Backup for Microsoft 365

 Let's explore how to restore M365 objects by using Veeam Backup for Microsoft 365. The backup data is from Archive Object Storage.

Video: Configure Backup Copy to Archive Storage on Veeam Backup For Microsoft 365

 Want to do 3-2-1 copy for Microsoft 365 backup?

Do check out the new capabilities on VB 365 v6 on how to configure Backup Copy Job to Archive Storage.