An important part of any Magento platform migration project is the process of migrating data. Moving from a platform like OS Commerce, WooCommerce, or AspDotNetStorefront to Magento may seem simple at first glance, but once you consider the type of data you need to import, the volume of data you need to move, and the timing required to successfully move your data, you’ll find that great care is required.
Data migration comes in three flavors: core data, custom EAV data, and purely custom data.
- Core data is, as you might expect, data that is required to make an entity function properly in Magento. The most common entities to be migrated include customers, catalog data like categories and products, and order/order item data.
- Magento has an EAV (Entity Attribute Value) system that allows developers to extend core data entities rather easily. For example, you may need to store a warehouse isle, shelf, and bin location for every product. While these fields are not available directly, these product attributes can easily be added via the Magento admin.
- The final set of data that impacts pricing is truly custom data. This is data that is not inherently supported by Magento and more often than not requires custom development. In the previous example, you may need to set the isle, shelf, and bin locations based on the region(s) the warehouse resides. This is not something that Magento can handle natively.
There are three key checkpoints that we need to account for when migrating data to include: initial data load, data validation and modification, and the go-live data load.
- The first key checkpoint is the initial data load. It is here that we take initial requirements, map source data to Magento entities, then perform an initial data load. Process-wise, while this takes place, the front-end team is usually working on the theme.
- The second checkpoint is more of a QA phase. It is at this time that we validate that data was loaded as required, determine whether all data points were captured during requirements, and if necessary make a second pass. During this phase, the back-end team is usually working on deliverables that require custom development.
- Next we prep for the final go-live data load. As a typical implementation can take between three and six months, there are a number of new customers, products, and orders that have been added to the source system since the initial snapshot was taken. Depending on the amount of data, we will either perform a differential (loading just the new and changed data) or a complete reimport.
There are typically three options for migrating data to include:
- Magento’s import process supplemented with 3rd party modules, data migration services, and agency support. Magento’s import process and 3rd party modules have a heavy reliance on CSV files. This can be a barrier for many clients that don’t have the ability to create CSV files from their source data. The source data may reside in an Oracle or Microsoft database, may only be accessible via Web Services, or might reside in other types of data files like XML or Excel. Even when the expertise is available to create CSV files, there is also the issue of performance. Importing hundreds of thousands of customers and orders can take many hours or even days. Because it can take so long to import large data sets, clients will often prune the data to a smaller set for testing purposes only to realize that the process fails on the final go-live data import!
- Using 3rd party services to migrate data often works well for core data and custom EAV data as the service can often take direction from the client. Where they typically fall short is when you need to import advanced data or large data sets. But the biggest hurdle is timing, specifically importing go-live data. The time to complete an import can range from a few days to a couple of weeks; this is not an option when go-live data imports are required. Another item that is not obvious or well known initially is that most services are located overseas. Your sensitive data ends up in the hands of an offshore operation leaving you no real legal recourse for any data theft, data loss, or migration problems.
- Last and most effective is ADaM-T™. We at Aydus™ have developed the Aydus Data Migration Tool. ADaM-T™ allows us to precisely, effectively, and quickly:
- Bring over all data independent of data source (including CSV, Excel, XML, Oracle, MySQL, Microsoft SQL, and web services to name but a few of the most common data sources).
- Perform transformations and lookups on the fly.
- Handle large datasets with ease.
- Work with both Magento Community and Enterprise editions.
- Have the entire data transaction take place in the United States by a US company.
ADaM-T™ is in fact one of our secret weapons. I’m often asked how we can provide so much more value than our competition and the difference, without exception comes down to our ability to import data at a low internal cost.
About the Author
Chris Coddington is a Partner at Aydus™ Consulting and has specialized in eCommerce as a Technical Lead for the last 10 years. To learn more about Chris, see his profile on LinkedIn (www.linkedin.com/in/coddington).If you need to help with your data, please don’t hesitate to reach out to Chris (firstname.lastname@example.org).
Aydus™ is a super creative design and digital agency with offices in Los Angeles, California and Bozeman, Montana that specializes in all things eCommerce. We are passionate about developing websites, integrations, and apps that not only convert but are intuitive, reliable, and user friendly. To learn more about Aydus™, email email@example.com.