Push vs. Pull Mechanisms in the Gateway

Push & Pull: Custom Integrations (formerly known as: Private Apps)

Custom Integrations are implemented directly into your own technology stack, allowing you to programmatically send (push) metadata updates to, or retrieve (pull) specific entity data from your Data Custodian profile held by the Gateway. These integrations follow both the Push and Pull models, where your system proactively informs the Gateway of changes within your data, and/or retrieves data we hold on your behalf for use within your own systems. In this setup, your system is the source of truth, meaning the Gateway will always rely on your data and restrict manual modifications to metadata within the web portal.

More information can be found here on how to create and manage a Custom Integration within your Gateway Data Custodian profile.

Pull: Predefined Integrations (formerly known as: Gateway Apps and Federated Metadata Automation (FMA))

Predefined Integrations operate within the Gateway itself, periodically retrieving metadata from an external source you manage. This follows a Pull model, where the Gateway requests and syncs data on a scheduled basis. As with Custom Integrations, Predefined Integrations remain an authoritative source on behalf of your catalogue’s source of truth. As with Custom Integrations, this setup ensures that metadata changes cannot be made manually within the web portal.

More information can be found here on how to create and manage a Predefined Integration within your Gateway Data Custodian profile. As well as guidelines on adopting Gateway Schemas here.

What should I use?

That depends! But, we recommend the following:

• Use a Predefined Integration if you already have a metadata catalogue and want automatic synchronisation and lack technical resource availability to create Custom Integrations.

• Use a Custom Integration if you want full control over data management, within your own technology stack.

What else can I do with Gateway APIs?

We’re not just limited to metadata onboarding within our APIs. Our APIs are available to all users of the Gateway who have data to share with the community, and can be used to administer your data presence on the Gateway. From Users all the way to Data Uses and Publications. Check out the here and if you ever need some guidance, feel free to reach out to us here.

It should be noted that Predefined and Custom Integrations can be used for all types of data eventually*. Not just Dataset Metadata. This means that in addition to specific endpoints offered within the APIs for Gateway held entities, the same can be used for many other data types we hold. Basically, the open APIs in conjunction with the Integrations can be used interchangeably depending on your need and development expertise.

*Currently, only Dataset metadata is supported through Predefined Integrations (formally known as: Gateway Apps). Support for further data types will be added soon. Please see the release notes/Changelog on the Gateway to understand when further supported data types are available.