Building A Better Connector Ecosystem: Design & Integration

Hey everyone, let's dive into how we're revamping our approach to integration modules for curated connectors. We're talking about building a robust ecosystem of connectors that are easy to use, maintain, and scale. This involves leveraging our existing workflow service steps to create a more user-friendly experience, similar to platforms like n8n, without the need for bespoke code for each vendor. This is a crucial move for the future of our platform, so let's get into the nitty-gritty!

The Current State of Affairs

Currently, we've got a solid foundation with our service steps. These steps already handle templated outbound HTTP calls. This includes essential features like secret resolution to keep your sensitive data safe, health gating to ensure services are running smoothly, retries to handle temporary hiccups, and response capture to grab the information you need. These components reside in services/core/src/workflowOrchestrator.ts, taking care of the low-level details, so you don't have to. We also have the ability to package integration logic inside AppHub modules, granting us tight control over the lifecycle and versioning of these crucial components. This is handled via services/core/src/moduleServices/runner.ts. This allows us to push updates, fix bugs, and add new features in a streamlined manner.

Now, the workflow builder, as it stands, presents raw request blocks. This is functional, but it lacks the polish and ease of use we're aiming for. We want to elevate the user experience, bringing it closer to that of user-friendly platforms. This ensures that creating integrations is a breeze, no matter your technical expertise. This higher-level UX will drastically improve the overall experience, making integrations a joy rather than a chore. We want our users to focus on the what and less on the how. So, let's build something awesome!

Proposal: The Path Forward

Here's the game plan: We're going to ship an "Integrations" module (or a cluster of domain-specific modules). These modules will expose service targets for outbound actions and job/service targets for inbound triggers. Think of these modules as the brains behind each connector. They'll know how to talk to various services, handling the complexity behind the scenes. They will speak the language of different APIs, taking on the burden for you. This modular approach ensures that we can evolve our offerings quickly and provide a smooth experience.

Next up, we'll define connector actions with typed inputs and outputs. This means that when you use a connector, you'll see clear fields for what you need to provide and what you can expect to get back. We'll also include default values and capability metadata. This allows the workflow builder to render curated steps, eliminating the need for users to wrangle with raw JSON. This leads to a more intuitive and guided experience, making it easier for anyone to build integrations.

We'll also be managing credentials through shared descriptors. This will include OAuth setups, API keys, and tenant IDs. Both the module and workflows can reference these credentials safely. Security is paramount, so we will implement robust practices to keep your data safe. The shared descriptors will ensure that sensitive information is handled securely and consistently across the platform. These descriptors will act as a single source of truth for credential management, streamlining the process and reducing the risk of errors.

We will retain service-registry entries as an escape hatch, mainly when customers need to host the integration themselves. However, module-backed services will be the default path. This gives us flexibility, allowing users to either leverage our managed integrations or host them independently, depending on their needs. The primary focus will be on the module-backed approach, as it provides the most consistent and managed experience.

Open Questions: Let's Discuss

We've got some crucial questions that need answering as we move forward.

• How do we model connector metadata so both the workflow UX and runtime can consume it without duplication? This is a design challenge that we need to address head-on. We want to ensure that the information about each connector is available in both the user interface and the system that runs the integrations, and this is all without duplicating effort. This design will make sure both sides of the integration are working in harmony. We want a system that is both efficient and effective, allowing us to evolve our integrations quickly without creating a headache.
• What minimal credential lifecycle (setup flows, token refresh) do we need before launch? Credentials are the keys to the kingdom, so we need a solid plan for managing them. We need to build out robust setups for the initial user, along with features that will automatically refresh tokens to maintain a user's access to third-party services. Making the user's life easier with easy and secure authentication is key. This will provide a safe and easy user experience.
• Which initial connectors/unified abstractions deliver the most value (e.g., email, storage, ticketing)? We want to start with the integrations that will have the biggest impact. Email, storage, and ticketing are all areas where robust integrations will make a difference. We will focus on building connectors that will provide the most immediate value to our users. This will ensure we are getting the most bang for our buck and will make sure users are getting the features they need most. Building value is paramount to the success of the project, and we will continue to deliver.

Next Steps: Action Time!

We're not just talking about this; we're going to do it! Here's what's on the horizon:

• Run a short design spike to nail down schema/descriptor formats and UX implications. We will be running design sprints to solidify our plans. We need to create a concrete, testable plan to guide the development. We'll be experimenting with different ways to model connector metadata, making sure the experience is intuitive and effective. This hands-on approach ensures we're building something that's both technically sound and user-friendly.
• Inventory near-term connectors and map them to module targets versus existing services. We need to take stock of what connectors are needed most and determine whether they'll fit into our new module-based system. We will evaluate each integration, assessing its current architecture and making the determination to move forward. This assessment will make sure that our strategy is optimized and will take advantage of the best approach for each integration. This will ensure that we're focusing our efforts on the most impactful integrations.
• Prepare developer docs describing how to build and publish new connector modules. We're not just building the infrastructure, we're also creating resources to empower developers to contribute. We will provide comprehensive documentation, outlining the steps required to build and publish new connector modules. This will open the door for our community to get involved, expanding our library of integrations and accelerating innovation. This collaborative approach will benefit everyone, expanding our capabilities and community involvement.

This is a huge step in making our platform even more robust and user-friendly. With these improvements, we can all look forward to a future where integrating different tools is a piece of cake. Stay tuned for more updates, and thanks for being a part of this journey!