We’re excited to announce Data Import 1.0! Data Import is a CSV to Ed-Fi API extract-transform-load (ETL) tool developed with a 360-degree view of the student in mind. We’ve designed this tool to help accommodate “long-tail” datasets that fall outside the preferred API integration path.
Many education-serving vendors, especially in the highly-requested SIS and Assessment domains, have provided real-time Ed-Fi API connections—which is ideal for data integration. With direct API integration, data flows seamlessly into ODS / API instances with a rich understanding of data integration and security patterns. Many more vendors are currently working to include Ed-Fi API connections in their products in these domains and others. Note: You can submit our Vendor Request Form to make sure your requests for vendor action are represented!
Why Data Import Matters
Data sources that do not have an active vendor or in-house supported, such as state reporting systems, may contain valuable data that could help paint a 360-degree view of student progress but there is no pathway to incorporate it into the ODS. We call data that falls into this category “long-tail.”
With legacy systems, there’s likely no technological pathway to incorporate into your Ed-Fi ODS / API, as many of these systems lack modern connectivity options and/or budget doesn’t allow for the necessary modernization.
Data Import will help agencies avoid these missed opportunities by providing for methods of loading legacy CSV data via the Ed-Fi API, still allowing for API security controls and other benefits such as “upserts” for data updates.
This pathway has already been helpful to Ed-Fi users such as Sherod Keen, Senior Application Support Analyst of the North East Florida Educational Consortium and President of FLCODE. Sherod shared, “Data Import supports all Ed-Fi domains, thus allowing the user to implement the entire Ed-Fi Data Standard.” This means connecting more data streams and providing a richer view of student progress and outcomes.
How Data Import Works
Data Import provides a mechanism to map legacy CSV files to the Ed-Fi Data Standard. Once a CSV file is mapped to the Ed-Fi Data Standard, Data Import provides methods to load them via the Ed-Fi API, aligning with security and data management benefits.
Data Import offers:
- CSV to Ed-Fi ODS / API mapping: An easy-to-use, intuitive web-based user interface to map CSV source data columns
Screenshot of Data Import’s Data Mapping Feature with Grand Bend 2019 Sample
- ETL Services: Command-line programs that load data based on mapping, translation to Ed-Fi descriptors via lookups and logging services to debug data loads
Screenshot of Data Import’s Logging Feature with Grand Bend 2019 Sample
- Template Sharing: We’ve heard from the Ed-Fi Community that the process of data mapping and loading is time-consuming and that it’s valuable if processes and reusable metadata can be shared amongst adopters. We’ve built a Template Sharing Service into Data Import to allow for the sharing of templates for loading similar legacy datasets. This includes Ed-Fi data mapping, value replacement lookups, and API metadata. Note: No actual result data is shared in this process.
Screenshot of Data Import’s Template Sharing Feature with Grand Bend 2019 Sample Templates
Ed-Fi adopter Mike Werner at the Center for Educational Leadership and Technology is particularly excited about this feature. “The exchange of the Data Import templates will allow the Ed-Fi Community to share their approaches, reducing development time and rework.” This is geared to help Ed-Fi adopters with similar data issues work together and reach the goal of reduced time and shared approach.
Data Import is now available for use! For full details and installation instructions, visit: https://techdocs.ed-fi.org/display/EDFITOOLS/Data+Import
