Handling integrations and complex software projects
One of the most common aspects of our work with clients is the need to integrate complex systems and services. Whether we are integrating data for analytics or financial reporting‚ or creating the infrastructure to support a system like Microsoft Dynamics or Salesforce‚ integration is an area full of challenges.
The complexity in integrations often stems from the need to reconcile disparate ways that systems can represent and interact with data (i.e. different formats). Fortunately‚ using standard reference data frameworks and industry standards can simplify integration workflows.
In this article we discuss the common types of software integration and some of the main challenges.
What are the different types of integrations?
Software integrations can be described as most types of integrations that allow you to hook up a piece of software or a service with another. There are lots of different types of software integrations which can be categorised as below:
Service to Service APIs
A service API has two parts: using services‚ and developing services. Service APIs are designed to allow developers to reuse services that they have created in other applications‚ or at other companies. By standardising the interface to these types of online resources‚ data can be pulled from multiple applications or websites more easily‚ without requiring the developer to rewrite the code.
Service APIs built on top of the cloud infrastructure enable you to make calls from a wide array of devices: voice (via speech recognition)‚ television‚ game consoles and set-top boxes‚ and mobile/smartphones. This means you can access your cloud services from anywhere‚ anytime with any kind of device.
Webhooks
Webhooks are the new buzzword in software automation. They’re a great way to automate mundane tasks‚ and they’re becoming an important part of every structured software stack‚ from software applications to courier logistics and even e-commerce sites. Webhooks are used by a number of web applications‚ as well as many enterprise solutions.
Webhooks are automated messages sent from apps when something happens. For example‚ a message is sent every time that an item is added to a list of your choosing. The message contains data relevant to the event (in this case‚ an array of objects with information about what was added) and is sent to a unique URL‚ essentially the app’s phone number or address.
Pre-prepared areas for data files to be exported
It is often the case that data files related to a site need to be exported on a regular basis. An example of this might be a website updating product inventory or page stats on a regular basis which needs to be processed with an analytics tool. As the medium for transferring files will commonly be an FTP server or similar network filesystem and it can be useful to have the ability to dump files there at a scheduled time. (For an FTP server to sync data‚ there needs to be a prearranged plan for files to be put in specific places on the server.)
Direct data source access
A common approach involves accessing a different system’s data store directly from your own services‚ bypassing the usual CRUD logic in favor of a faster path to the data and higher flexibility.
Accessing another database directly is often not the most ideal option for a number of reasons‚ but it does allow you to quickly and easily get your hands on some data. This might be helpful if you need dummy data‚ or just want to test a specific query without having to set-up a database or an entire application.
Accessing another database directly is also a pragmatic choice when faced with tight deadlines and other restrictions that make it difficult to implement any of the alternatives discussed above.
What are the main challenges encountered when working on complex system integrations?
One of the biggest challenges when working on complex systems integrations is that often we are integrating with systems that have no predefined way of integrating with anything else. Often‚ the systems we integrate with are closed‚ bespoke systems that require a third party to add API capabilities. Examples include Bespoke software and ERP (Electronic resource planning) systems. In addition to this‚ the data which is available‚ usually isn’t documented or maintained anywhere.
A lot of systems also do not have a staging integration so we’re building against production data‚ which can be rather risky as any changes will affect a live system. Ideally we like to have access to a staging environment to test any integrations before they are deployed to a live environment.
In the enterprise space‚ we see businesses with thousands of different systems across multiple departments. It can sometimes seem rather overwhelming to test and deploy changes across these various systems. Some organisations may have a dedicated team for managing changes but it could be that any changes will be made by a development team as part of their day-to-day work. This can create some interesting scenarios on how we test and deploy changes across the organisation or system environments.
What is the correct integration strategy to follow?
As data volumes continue to grow for any business‚ the challenges of storing and processing data become larger and more opaque.
This means that the right data integration strategy is essential for your company’s success. If you are trying to assess the best solution for your company’s needs‚ then your business model and overall data storage strategy should be strictly defined well ahead of any integration efforts.
From the outset‚ there should be a clear understanding of what data is stored where and where the master copy of this data is located.
We also recommend that you always have a production replica containing dummy data to test against. One of the best ways to fill your staging database is by creating data factories to simulate similar content while keeping production records safe and separate from test environments where they may be accidentally emailed or the potential to breach GDPR.
What is an API Integration?
An API integration is the connection between two or more applications‚ via their APIs‚ that lets those systems exchange data. The integration of multiple applications can help create a seamless experience for your mobile and web app users‚ better information sharing between departments and streamline repetitive tasks.
For example‚ using an API integration‚ when an airline finds out that a ticket has been purchased with a gift card‚ the airline can directly send a notification to the gift card company with some basic information about the purchaser. This way‚ the gift card company can notify the buyer or send them an email offer.
The challenges of working with third-party APIs
Whether you are starting a new project or looking to expand an existing one‚ relying on third-party APIs is a common practice. This can help you scale your business and save on development costs‚ but it also comes with some risks.
One of the most significant ones is that many third-party APIs aren’t as well documented as your own internal APIs or bespoke APIs that have been custom-built for your company. This can add time and expense to any project where your business relies on one or more third-party APIs.
One issue we often encounter is that documentation is limited to 200 status codes‚ telling you what’s happening when things are working. As soon as you get a 400 Bad Request or 500 Internal Server error code‚ you’re out on a limb without sufficient documentation to swiftly debug any issues. Again‚ this all adds to timescales and development costs.
You also can’t always depend on third-party APIs to be available‚ from down servers to unexpected closures‚ availability is something you need to constantly plan for.
How can APIs address legacy system challenges?
Businesses typically maintain large amounts of data which is stored and processed in different sources. These sources are usually old legacy systems. This information is locked in these applications and is very expensive to change or integrate with other pieces of core business systems through traditional means.
One way businesses are achieving this integration is through building new APIs. They can provide a secure integration layer for legacy systems by helping to streamline operations and address changes in this landscape such as new compliance requirements‚ easier data access among IT and business users‚ and external auditing.
APIs can also be used to transfer data between old systems and new systems and allow for the systems to change‚ or even be fully updated‚ behind the scenes. This reduces any interruption to the end-user and also reduces the inherent risks associated with an “all in one” system transfer.
Looking for a delivery partner with over 10+ years of experience in integrating complex systems and APIs? Read our case studies or get in touch today.
