Scaling Vortexa Oil & Gas Datasets with AWS Data Exchange

Thursday, 16 January, 2020

AWS Data Exchange has been born as a centralised easy-to-use way for the sellers to offer their datasets up for sale and the buyers to easily browse available datasets and use the same method to access any datasets. Simplistically speaking, it is a way to publish one or more datasets with one or more revisions of data in it in a way that provides multiple benefits to the sellers and buyers:

• Sellers: streamlined billing, additional marketing opportunities, access to a broader customer base that otherwise may have been difficult to reach, a scalable way to programmatically export terabytes of data once and have it downloaded multiple times without any extra costs or infrastructure overheads.

• Buyers: a unified interface to access the data, commitment from Amazon to develop and extend that interface, high data availability and deterministic latency, likely proximity of data to the computing environment in case a customer is using AWS for their modelling, as well as a trustworthy and familiar billing partner, Amazon.

Both benefits can reduce the seller costs and improve buyer user experience immensely compared to a home-grown bespoke system dedicated to sharing the raw data.

Here is how one of Vortexa's products looks like on AWS Marketplace:

There are a few useful articles that describe how to access the data programmatically. Fundamentally, accessing the data is as simple as (from an article on AWS Big Data Blog):

1. Configuring your prerequisites: an S3 bucket for your data and IAM permissions for using AWS Data Exchange.

2. Subscribing to a new Vortexa data product in AWS Data Exchange.

3. Setting up automation using Amazon CloudWatch events to retrieve new revisions of subscribed data products in AWS Data Exchange automatically.

Once that is done, the data in CSV format, in Vortexa's case, will automatically appear and get updated every time Vortexa publishes a new revision. You can then import and process it any way you like! The newer versions of Boto, a popular AWS SDK for Python, support AWS Data Exchange as well.

So in conclusion, AWS Data Exchange provides the buyers with an easy-to-use unified way to access multiple data sources, while creating an effective and efficient way to distribute data at scale for the sellers. It is a true win-win!

Author: Maksym Schipka, CTO at Vortexa

For more practical tips on Data Science and Tech - follow the VorTECHsa team on Medium