News Categories
Announcement (9) Amy Babinchak (64) Tips (1) SBS 2011 (6) Windows Essentials 2012 (4) Edwin Sarmiento (28) SQL Server (22) SQL Server 2012 (6) SQL Server Clustering (3) SQL Server Disaster Recovery (6) Windows Server 2008 Clustering (1) log shipping (1) Brian Higgins (3) Uncategorized (42) Hyper-V (67) Virtualization (13) Windows 8 (13) Cisco VPN Client (1) Windows Server 2012 (24) Friend of TT (4) Hangout (2) Office365 (4) DNS (8) Jeremy (7) Cliff Galiher (3) Active Directory (12) ClearOS (4) Linux (4) presentations (2) SQL PASS (6) Chris Matthews (4) Printers (2) SharePoint (8) SQL Server Administration (7) Windows PowerShell (3) recovery model (1) sql server databases (1) Dave Shackelford (7) SMB Nation (1) Steve (1) Boon Tee (5) Kevin Royalty (3) Lee Wilbur (2) Philip Elder (10) SMBKitchen Crew (31) Susan Bradley (15) AlwaysOn (1) AlwaysOn Availability Groups (4) readable secondaries (1) row versioning (1) undocumented (1) The Project (2) Webinar (3) Enterprise for SMB Project (9) Security (25) Remote Desktop Connection for Mac (1) Remote Desktop Services (8) Windows Server 2008 (1) Exchange (15) Powershell (6) Microsoft (15) Performance (7) data types (1) Server 2012 (1) monitoring (1) DevTeach (1) SQL Server High Availability and Disaster Recovery (5) Clusters (44) Hyper-V Server 2012 (2) Business Principles (26) Cost of Doing Business (13) DHCP (7) sbs (15) Windows Server (30) SMBKitchen (26) Windows Server 2008 R2 (4) StorageCraft (1) P2V (1) ShadowProtect (6) StorageCraft ShadowProtect (1) VHDs (1) Intel RAID (2) Intel Server System R2208GZ (1) Intel Server Systems (17) RAID (2) SAS (2) SATA (2) Server Hardware (12) Microsoft Licensing (2) OEM (2) System Builder Tips (4) Intel (5) Intel Channel Partner Program (4) Intel Product Support (10) Intel Server Boards (2) Intel Server Manager (2) Cloud (26) IT Solutions (2) On-Premises (20) SMB (9) WIndows Azure (2) StorageSpaces (1) Error (47) Error Fix (35) Intel Desktop Boards (2) Intel SSDs (2) SSD (2) Business Opportunity (17) Data Security (11) Identity Security (7) Information Security (14) Privacy (2) Intel Modular Server (6) Promise (2) Storage Systems (9) Live ID (2) Microsoft ID (4) User Profiles (2) Articles (2) Building Client Relationships (6) DBCC IND (2) DBCC PAGE (2) filtered indexes (2) SQL Server Index Internals (2) training (11) Adobe (3) Internet Street Smart (8) Intel Storage Systems (2) LSI Corp (2) LSI SAS6160 Switch (2) Storage Spaces (7) Firmware Update (2) Product Support (7) Hybrid Cloud Solutions (3) Server Core (2) MAXDOP (1) SharePoint 2013 (1) SharePoint best practices (1) SQL Server Authentication (1) Family (5) Alternatives (1) SBS 2011 Standard (4) Microsoft Small Business Specialist Community (2) Microsoft Surface (2) SBSC (2) Networking (4) Availability Groups (3) CANITPro (1) HA/DR (1) Step-By-Step: Creating a SQL Server 2012 AlwaysOn Availability Group (1) webcast (1) VMWare (2) Conferences (2) Client Focus (2) Disaster Recovery (6) Error Workaround (8) Troubleshooting (4) Logitech (2) Product Review (7) Windows Features (4) XBox Music (2) SBS 2008 All Editions (4) MDOP (2) Microsoft Desktop Optimization Pack (2) Software Assurance (2) W2012E (6) Windows Server 2012 Essentials (6) Internet Explorer (3) USB 3.0 (2) USB Hard Drive (2) Bug Report (2) Microsoft Office 365 (5) sharepoint online (2) BitLocker (2) Windows (2) Microsoft Update (3) Swing Migration (2) Windows Update (4) Outlook (2) Group Policy (9) WS2012e (2) WSUS (3) Office (3) Microsoft Downloads (5) Microsoft Office (3) DRP (3) Virtual Machines (2) Virtual Server Hardware (2) online course (1) SQL Server learning (7) 2 Factor Authentication (2) 2FA (2) PASS Summit 2013 (4) SQLPASS (5) Contest (1) e-learning (1) Udemy (1) smbtechfest (1) backups (2) PASS Summit First Timers (3) IIS (2) RD Gateway (4) RD RemoteApp (2) RDWeb (4) Remote Desktop Connection (2) Remote Web Access (2) Remote Web Workplace (2) Cryptolocker (6) Backup (4) Restore (2) CryptoLocker (1) AuthAnvil (1) SBS 2003 (1) SBS Migration (1) Windows Server 2012 R2 (9) Documentation (1) IE 11 (4) testimonials (11) SQL Server 2008 (1) Best Practices (1) Support (1) Intel Xeon Processor (1) RemoteApp (1) Android (1) iOS (1) Hyper-V Replica (2) PowerShell (2) SBS (3) Break (1) Business Intelligence (1) Excel 2013 (1) Power Map (1) Power Query (1) PowerBI (1) MultiPoint (2) Surface (1) Net Neutrality (1) Opinion (2) ASP (9) HP (2) Scale-Out File Server (8) SOFS (10) Windows Phone (1) Updates (1) Intel NUC (1) Intuit (1) QuickBooks (1) Office364 (1) Intel Server Systems;Hyper-V (1) Firewall (1) Patching (1) Mobile (1) Mobility (1) sharepoint (1) Microsoft Security (1) Beta (1) Storage Replication (1) outlook (1) Hyper-V Setup (3) JBOD (1) Azure (1) PCI (1) PCI DSS (1) PII (1) POS (1) MicroStaff (2) Catherine Barr (2) Third Tier (1) BeTheCloud (1) BrainExplosion (1) LookAWhale (1) Manuel (1) Rayanne (3) SuperSecretNews (1) TechYourBooks (3) Managed Services (1) Training (1) E-mail (1)
RSS Feed
Microsoft turns data storage upside down
Posted by Amy Babinchak on 07 August 2018 05:01 PM

Understanding how SharePoint and OneDrive for Business are related

SharePoint and OneDrive for Business are linked. SharePoint is the data storage location and OneDrive for Business is the client that manages the sync process. That part is pretty easy to understand. But to confuse the matter, Microsoft gave OneDrive for Business the user’s own private storage space, which although it is stored in SharePoint does not draw from your SharePoint storage quota.

You can think of the OneDrive for Business personal storage location as the user folder from the on-premises world. SharePoint is the storage location where these “user folders” reside and can be thought of as the equivalent of the server from the on-premises world. “User folders” do not take up any of your SharePoint organizational quota.

In addition to syncing and storing your own private files, the OneDrive for Business client can also sync corporate data stored elsewhere in SharePoint. So this client provides access to files in both locations. Best of all you can choose what you “see” in your OneDrive for Business client and what you are going to sync locally to your computer.

To muddy the waters just a bit more, Microsoft recently announced that One Drive for Business will soon start to offer the option to automatically sync your local profile default data locations such as the documents and pictures folders. And it will also have one-button ransomware protection for your files.

So now we’re storing personal data, bits of the user profile, and we’re syncing locally some or all of the data in SharePoint. But we’re still upside down from how business has historically stored data because our corporate space is smaller than the personal space. Basically, if you want to store all of your data in Office 365, then you’ve got some reorganizing to do and some educating of your staff to do so they know where to store things now.

Thinking it through

Knowing that we have more space for private files than we have for general corporate data means that we have to think about how data is going to be stored in the cloud. Or you could purchase more SharePoint data storage space and not think about it. But let’s see how we might rethink data storage and convert it into the cloud model from the on-premises model of data storage.

To do this we’re going to first fix up our data to make sure that we have naming conventions that will be accepted in the cloud. Then we’ll look at archiving. Finally, we’re going to take a look at who really needs access to files and think about how modern applications and the cloud might mean we can organize them differently.

Ready to migrate files into OneDrive for Business?

You‘ll need to be aware of a few limitations when deciding to migrate your files into OneDrive for Business. The biggest gotchas for my clients have been file-naming conventions and total character length. But there’s also a file size cap and a few file types that aren’t allowed too. So you might need to do some data massaging before you migrate.

Here’s what you need to know:

These are the characters that aren’t allowed in your file names: <, >, :, ", |, ?, *, /, \

These are the file names that aren’t allowed: Icon .lock CON PRN AUX NUL COM1 COM2 COM3 COM4 COM5 COM6 COM7 COM8 COM9 LPT1 LPT2 LPT3 LPT4 LPT5 LPT6 LPT7 LPT8 LPT9.

Any filename starting with ~$ or desktop.ini and anything with this string of characters _vti_.

These are the folder names that are not allowed: _t _w _vti_ and forms when it is at the root level.

Each file must be less than 15GB in size, which, honestly, should never be a problem. This is data file storage after all, not database storage.

The total file path must be under 400 characters. This one is likely to catch many people.

Fixing file names

I can’t do a better job at providing a smooth easy solution for fixing the file-naming conventions than Nik D’Agostino, product marketing manager at Lowry Solutions, has in his fabulous article on LinkedIn. So I’ve pulled this information from his article for you.

1) Download the Bulk Rename Utility Tool and extract it.

2) Uncheck all of the group except for 3 and 12.
data storage
3) Make sure folders, files and subfolders are selected under group 12.
data storage
4) Fill out group 3 with the characters you want to find and replace. I recommend replacing each of the following characters \ / : * ? " < > | # % with a dash or space.

5) Navigate to the folder whose contents you want to rename (in this case the folder we copied to our desktop) in the left window pane then make sure you select all of the files, folders, and subfolders you want to rename by selecting them in the right window pane and click the Rename button.

6) Repeat this rename process individually for each of the following invalid characters: \ / : * ? " < > | # %.
data storage

Archive your data

Many businesses are carrying around a lot of data that they really probably don’t need but can’t bear to part with. In my experience, this actually makes up the bulk of data currently sitting on servers. When hard drives got cheap, data volumes went up. Because we’re moving to the cloud it might not make sense to take all of the legacies forward with us. This might be a hard sell but consider leaving some of it behind.

You have a couple of options for this:

  1. Archive the oldest files permanently onto external disks and file them away.
  2. Archive the files you probably won’t need but can’t part with just yet into an Azure file store location or purchase additional SharePoint space and put them into an archive document library.

Azure offers SMB shares to file storage locations. Since these are archived files you are thinking that you probably won’t need you can just map a few people to the SMB share. The cost of SMB file storage in Azure is pretty reasonable. It will cost you around $.10 per GB plus some small transaction costs but it will be quickly accessible and you can map a drive to it which is incredibly convenient.

Microsoft also offers Block Blob archive storage for $.002 per GB, but if you need to read it, be aware that you have to pull the entire blob out of the archive for around $26 for the operation and it will take a number of hours before it begins (as many as 15 hours). If you truly just want to store the data for the long term, $.002 is the least expensive way to do it.

The other option is to purchase additional space for SharePoint. This simply expands your data storage space in SharePoint and you can distribute it among sites and libraries however you like. But this is the most expensive option at $.20 per GB.

Start the discussion

Now that you know what the costs are going to be, it’s time to determine how much of your data is actually archive data. Azure says that archive data must not have been accessed for at least 180 days. But I’ll guess that most businesses have data that hasn’t been accessed for 180 days, one year, two years, or even five years. You’ve probably not looked at your data in this way before, but now it is the time.

Back in 2014, the Scripting Guy wrote up a simple script called Get-Neglected files that uses PowerShell to gather a list of files that haven’t been written to since period of time that you define. I recommend using this method. He uses the file property LastWriteTime to determine when the last time the file changed which is exactly what you’ll be after when determining which of your files are truly archive material.

Use the | to export the data into a CSV file where you can then calculate the amount of drive space that you’re going to need for your archive and for your working data.

Reevaluating folder depth for cloud compatibility

Now for the hard part. Deep complex folder/file structures don’t work very well in the cloud. The website rule of thumb that people won’t click more than twice to get to something very nearly applies to cloud-stored files too. Yes, it’s a whole new world. Yes, this means changing the way that businesses think of their data. The benefit of this exercise is that it is also going to expose teams that you didn’t realize existed in your organization, even though they don’t think of themselves in that way. As you go through and look at the folders, see who has access to them and who is actually using them to discuss whether the structure needs to be as deep and complex as it is. You’re going to find that most folders are accessed by just a few people and those people are entering the folder structure at different points to avoid the click, click, click, click drill-down process. We want to expose those points as they are logical places to break the chain. Further, you’re also going to expose areas of your folder structure that are only used by one person. Those should be moved into their OneDrive for Business personal storage location.

More motivation for simple folder structures

The limitation that my clients have the toughest problem staying under is the 400-character file path limit. Remember that your file path isn’t just the folder depth but that it also includes the SharePoint online URL too. For many businesses, this will mean a reevaluation of their folder structure to make it suitable for cloud storage and will give you some leverage when talking to staff about reevaluating how they are storing and naming files.

This can be a painful process, but is it a bad thing? I don’t think so. Often the current folder structure was grown on-premises over a long period of time and as Microsoft Office began to support longer and longer character limits file and folder names got longer too. The information explosion has also caused workers to give files descriptive names which are also longer. So we have some sacred cows to deal with during this migration. You are going to get a lot of pushback and unwillingness to sit down and hash through this process. But the end result will be worth it. A shorter pathed flatter file/folder structure is much easier to navigate on mobile devices. Since your cloud files will end up being viewed not only in OneDrive but also Microsoft Teams, SharePoint and other Office 365 applications, people will find that a flatter structure benefits everyone.

Migrating the data

Microsoft has produced a great migration tool for getting your data from on-premises and into SharePoint. You can read about it here. It allows you to pick a folder from your server and populate it into the SharePoint library of your choice.
It is very interesting to note that Microsoft recommends standing up several virtual machines to support the data transfer process. This will let you get multiple upload streams going at once. Take note, too, of the upload speeds. This is probably not something that you’re going to accomplish over a single weekend.

Type of metadata Examples Average customer experience
Light ISO files, video files 2 TB/day
Medium List items, Office files (~1.5MB) 1 TB/day
Heavy List items with custom columns, small files (~50kb) 250 GB /day

Data storage bottom line: Think it through before you go

The actual data move is going to be the least of your problems. In this case, the real work is all in the data preparation and getting the business truly ready for a move into the cloud. It’s your skill at consulting and working through internal politics that is going to make or break this project. Microsoft has turned data storage thinking on its head by providing huge personal storage and small corporate storage with their plans. If you want to make that work and utilize the included storage, then you’ll have some work to do.


About Third Tier

Open a ticket with us! Established in 2008, Third Tier only works for IT Professionals by providing them with access to advanced support services. No one can know it all these days, so we give IT pros a place to go to get the hands on support they need in areas they normally don’t work in or problems they’ve never encountered. We also work on projects, fix their accounting practices and do many, many migrations and other installations. Our staff covers a wide range of technologies.




Comments (0)
Post a new comment
Full Name:
CAPTCHA Verification 
Please enter the text you see in the image into the textbox below. This is required to prevent automated registrations and form submissions.

Help Desk Software by Kayako Fusion