# Create Catalog POST /api/v1/catalogs # Update Catalog PUT /api/v1/catalogs/{id} # Check Connection POST /api/v1/connector_definitions/check_connection # Connector Definition GET /api/v1/connector_definitions/{connector_name} # Connector Definitions GET /api/v1/connector_definitions # Create Connector POST /api/v1/connectors # Delete Connector DELETE /api/v1/connectors/{id} # Connector Catalog GET /api/v1/connectors/{id}/discover # Get Connector GET /api/v1/connectors/{id} # List Connectors GET /api/v1/connectors # Query Source POST /api/v1/connectors/{id}/query_source # Update Connector PUT /api/v1/connectors/{id} # Introduction Welcome to the AI Squared API documentation! You can use our API to access all the features of the AI Squared platform. ## Authentication The AI Squared API uses a JWT-based authentication mechanism. To access the API, you need a valid JWT token which should be included in the header of your requests. This ensures that your interactions with the API are secure and authenticated. ```text --header 'Authorization: Bearer ' ``` It is advised to keep your JWT token safe and not share it with anyone. If you think your token has been compromised, you can generate a new token from the AI Squared dashboard. ## API Endpoints The AI Squared API is organized around REST. Our API has predictable resource-oriented URLs, accepts JSON-encoded request bodies, returns JSON-encoded responses, and uses standard HTTP response codes, authentication, and verbs. ### Base URL The base URL for all API requests is `https://api.squared.ai/api/v1/` ### API Reference The API reference contains a list of all the endpoints available in the AI Squared API. You can also use the navigation bar on the left to browse through the different endpoints. Models are the core of the AI Squared API. They represent the different entities in the AI Squared platform. Connectors help connect various data warehouse sources or destinations to the AI Squared platform. Syncs help you sync data between different data warehouse sources and destinations. Audiences allow you to send targeted customer segments from data sources to various destinations. ## Pagination Requests that return multiple items will be paginated to 100 items by default. You can specify further pages with the `page` parameter. You can also set a custom page size up to 100 with the `page_size` parameter. ```text https://api.squared.ai/api/v1/models?page=2&page_size=50 ``` ## Rate Limiting The AI Squared API is rate limited to 100 requests per minute. If you exceed this limit, you will receive a `429 Too Many Requests` response. ## Errors The AI Squared API uses conventional HTTP response codes to indicate the success or failure of an API request. In general, codes in the `2xx` range indicate success, codes in the `4xx` range indicate an error that failed given the information provided, and codes in the `5xx` range indicate an error with AI Squared's servers. # Create Model POST /api/v1/models # Delete Model DELETE /api/v1/models/{id} # Get Models GET /api/v1/models # Get Model GET /api/v1/models/{id} # Update Model PUT /api/v1/models/{id} # Get Report GET /api/v1/reports # List Roles GET /enterprise/api/v1/roles Retrieves a list of all roles available. # List Sync Records GET /api/v1/syncs/{sync_id}/sync_runs/{sync_run_id}/sync_records Retrieves a list of sync records for a specific sync run, optionally filtered by status. # Sync Run GET /api/v1/syncs/{sync_id}/sync_runs/{sync_run_id} Retrieves a sync run using sync_run_id for a specific sync. # List Sync Runs GET /api/v1/syncs/{sync_id}/sync_runs Retrieves a list of sync runs for a specific sync, optionally filtered by status. # Create Sync POST /api/v1/syncs # Delete Sync DELETE /api/v1/syncs/{id} # List Syncs GET /api/v1/syncs # Manual Sync Cancel DELETE /api/v1/schedule_syncs/{sync_id} Cancel a Manual Sync using the sync ID. # Manual Sync Trigger POST /api/v1/schedule_syncs Trigger a manual Sync by providing the sync ID. # Get Sync GET /api/v1/syncs/{id} # Get Sync Configurations Get /api/v1/syncs/configurations # Test Sync POST /enterprise/api/v1/syncs/{sync_id}/test Triggers a test for the specified sync using the sync ID. # Update Sync PUT /api/v1/syncs/{id} # Data Apps Data App Image Watch out for this space, **Data Apps** coming soon! # Models AI/ML Models in AI Squared allows users to define how to gather the required data for input variables, for the AI/ML models they connect to, using AI/ML Source. Hero Light ## AI/ML Model Creation Users can select an existing **AI/ML Source Connector** and use predefined **harvesting strategies** to dynamically populate input variables. This harvesting process is especially useful for utilizing **Data Apps** through no-code integration. After the AI/ML Model is created, users can develop visualizations and create **Model Cards** via Data Apps for further insights and analysis. *** ## Harvesting Harvesting retrieves input parameters from business tools, essential for real-time machine learning model execution. Currently, we support harvesting input variables from sources such as **CRM systems** (e.g., Salesforce, Dynamics 365) and **custom web applications**. #### Harvesting Strategies: * **DOM (Document Object Model) Element Extraction**: Harvest data from specific web page elements. * **Query Parameter Extraction**: Extract data from URL query parameters. #### Data Types Supported: * **Text**: Such as Customer names, product descriptions, etc. * **Images**: Used for image recognition. * **Dynamic Variables**: Values that change with user interactions. ### Integration with Model Creation Harvesting is integrated into the model creation process, allowing users to define what data should be collected, ensuring real-time processing during model invocations. *** ## Preprocessing **Preprocessing** is an important step that occurs after data harvesting and before model inference. It transforms and formats the harvested data to meet the specific input requirements of the machine learning model. ### Key Preprocessing Tasks: * **Parsing Specific Attributes**: Extract and format the required data fields. * **Resizing Images**: Ensure images are resized appropriately for image processing models. These preprocessing steps ensure the data is properly prepared, optimizing accuracy and efficiency during real-time model inference. *** # Sources AI/ML sources help users connect to and retrieve predictions from hosted AI/ML model end points. These sources are critical for integrating predictive insights from AI/ML models into your workflows, enabling data-driven decision-making. Source Image We support a wide variety of AI/ML models as data sources. To get started, simply click on the specific tabs below for setup guides, to connect an AI/ML source of your choice. *** Once the source is created, it will be displayed in the **Sources** tab under **AI/ML Sources** category to easily differentiate it from other types of sources. # Core Concepts The core concepts of AI Squared are the foundation of your data journey. They include Sources, Destinations, Models, and Syncs. Understanding these concepts is crucial to building a robust data pipeline. Hero Light ## Sources: The Foundation of Data ### Overview Sources are the starting points of your data journey. It's where all your data is stored and where AI Squared pulls data from. Hero Light These can be: * **Data Warehouses**: For example, `Snowflake` `Google BigQuery` and `Amazon Redshift` * **Databases and Files**: Including traditional databases, `CSV files`, `SFTP` ### Adding a Source To integrate a source with AI Squared, navigate to the Sources overview page and select 'Add source'. ## Destinations: Where Data Finds Value ### Overview 'Destinations' in AI Squared are business tools where you want to send your data stored in sources. Hero Light These can be: * **CRM Systems**: Like Salesforce, HubSpot, etc. * **Advertising Platforms**: Such as Google Ads, Facebook Ads, etc. * **Marketing Tools**: Braze and Klaviyo, for example ### Integrating a Destination Add a destination by going to the Destinations page and clicking 'Add destination'. ## Models: Shaping Your Data ### Overview 'Models' in AI Squared determine the data you wish to sync from a source to a destination. They are the building blocks of your data pipeline. Hero Light They can be defined through: * **SQL Editor**: For customized queries * **Visual Table Selector**: For intuitive interface * **Existing dbt Models or Looker Looks**: Leveraging pre-built models ### Importance of a Unique Primary Key Every model must have a unique primary key to ensure each data entry is distinct, crucial for data tracking and updating. ## Syncs: Customizing Data Flow ### Overview 'Syncs' in AI Squared helps you move data from sources to destinations. They help you in mapping the data from your models to the destination. Hero Light There are two types of syncs: * **Full Refresh Sync**: All data is synced from the source to the destination. * **Incremental Sync**: Only the new or updated data is synced. # DBT Models Watch out for this space, **DBT Modelling** coming soon! # Overview ## Introduction **Models** are designed to define and organize data, simplifying the process of querying from various sources. Hero Light This guide outlines the process of creating a model, from selecting a data source to defining the model using various methods such as SQL queries, table selectors, or dbt models. ## Understanding the Model Creation Process Creating a model in AI Squared involves a series of steps designed to streamline the organization of your data for efficient querying. This overview will guide you through each step of the process. ### Step 1: Navigate to the Models Section To start defining a model: 1. Access the AI Squared dashboard. 2. Look for the `Define` menu on the sidebar and click on the `Models` section. ### Step 2: Add a New Model Once you log in to the AI Squared platform, you can access the Models section to create, manage, and monitor your models. 1. Click on the `Add Model` button to initiate the model creation process. 2. Select SQL Query, Table Selector, or dbt Model as the method to define your model. ### Step 3: Select a Data Source 1. Choose from the list of existing connected data warehouse sources. This source will be the foundation for your model. Hero Light ### Step 4: Select a Modeling Method Based on your requirements, select one of the following modeling methods: 1. **SQL Query**: Define your model directly using an SQL query. 2. **Table Selector**: For a straightforward, visual approach to model building. 3. **dbt Model**: Ideal for advanced data transformation, leveraging the dbt framework. Hero Light ### Step 5: Define Your Model If you selected the SQL Query method: 1. Write your SQL query in the provided field. 2. Use the `Run Query` option to preview the results and ensure accuracy. Hero Light ### Step 6: Finalize Your Model Complete the model setup by: 1. Adding a name and a brief description for your model. This helps in identifying and managing your models within AI Squared. Hero Light #### Unique Primary Key Requirement * **Essential Configuration**: Every model in AI Squared must be configured with a unique primary key. This primary key is pivotal for uniquely identifying each record in your dataset. * **Configuring the Primary Key**: * During the final step of model creation, select a column that holds unique values from your dataset. Ensuring the uniqueness of this primary key is crucial for the integrity and accuracy of data synchronization. * **Importance of a Unique Key**: * A unique primary key is essential for effectively managing data synchronization. * It enables the system to track and sync only the new or updated data to the designated destinations, ensuring data consistency and reducing redundancy. After completing these steps, your model will be set up and ready to use. # SQL Editor SQL Editor for Data Modeling in AI Squared ## Overview AI Squared's SQL Editor allows you to define and manage your data models directly through SQL queries. This powerful tool supports native SQL commands compatible with your data warehouse, enabling you to seamlessly model your data. ## Creating a Model with the SQL Editor ### Starting with a Query Begin by writing a SQL query to define your model. For instance, if using a typical eCommerce dataset, you might start with a query like: ```sql SELECT * FROM sales_data.customers ``` ### Previewing Your Data Click the `Preview` button to review the first 100 rows of your data. This step ensures the query fetches the expected data. After verifying, proceed by clicking `Continue`. **Important Note:** The model cannot be saved if the query is incorrect or yields no results. ### Configuring Model Details Finalize your model by: * Naming the model descriptively. * Choosing a column as the Primary Key. ### Completing the Setup Finish your model setup by clicking the `Finish` button. ## Unique Primary Key Requirement Every model requires a unique primary key. If no unique column exists, consider: * Removing duplicate rows. * Creating a composite column for the primary key. ## Handling Duplicate Data To filter duplicates, use a `GROUP BY` clause in your SQL query. For instance: ```sql SELECT * FROM customer_data GROUP BY unique_identifier_column ``` ## Composite Primary Keys In scenarios where a unique primary key is not available, construct a composite key. Example: ```sql SELECT customer_id, email, purchase_date, MD5(CONCAT(customer_id, '-', email)) AS composite_key FROM sales_data ``` ## Saving a Model Without Current Results To save a model expected to have future data: ```sql UNION ALL SELECT NULL, NULL, NULL ``` Add this to your query to include a dummy row, ensuring the model can be saved. ## Excluding Rows with Null Values To exclude rows with null values: ```sql SELECT * FROM your_dataset WHERE important_column1 IS NOT NULL AND important_column2 IS NOT NULL ``` Replace `important_column1`, `important_column2`, etc., with your relevant column names. # Table Selector Watch out for this space, **Visual Table Selector** coming soon! # Incremental - Cursor Field Incremental Cursor Field sync transfers only new or updated data, minimizing data transfer using a cursor field. ### Overview Default Incremental Sync fetches all records from the source system and transfers only the new or updated ones to the destination. However, to optimize data transfer and reduce the number of duplicate fetches from the source, we implemented Incremental Sync with Cursor Field for those sources that support cursor fields #### Cursor Field A Cursor Field must be clearly defined within the dataset schema. It is identified based on its suitability for comparison and tracking changes over time. * It serves as a marker to identify modified or added records since the previous sync. * It facilitates efficient data retrieval by enabling the source to resume from where it left off during the last sync. Note: Currently, only date fields are supported as Cursor Fields. #### #### Sync Run 1 During the first sync run with the cursor field 'UpdatedAt', suppose we have the following data: cursor field UpdatedAt value is 2024-04-20 10:00:00 | Name | Plan | Updated At | | ---------------- | ---- | ------------------- | | Charles Beaumont | free | 2024-04-20 10:00:00 | | Eleanor Villiers | free | 2024-04-20 11:00:00 | During this sync run, both Charles Beaumont's and Eleanor Villiers' records would meet the criteria since they both have an 'UpdatedAt' timestamp equal to '2024-04-20 10:00:00' or later. So, during the first sync run, both records would indeed be considered and fetched. ##### Query ```sql SELECT * FROM source_table WHERE updated_at >= '2024-04-20 10:00:00'; ``` #### Sync Run 2 Now cursor field UpdatedAt value is 2024-04-20 11:00:00 Suppose after some time, there are further updates in the source data: | Name | Plan | Updated At | | ---------------- | ---- | ------------------- | | Charles Beaumont | free | 2024-04-20 10:00:00 | | Eleanor Villiers | paid | 2024-04-21 10:00:00 | During the second sync run with the same cursor field, only the records for Eleanor Villiers with 'Updated At' timestamp after the last sync would be fetched, ensuring minimal data transfer. ##### Query ```sql SELECT * FROM source_table WHERE updated_at >= '2024-04-20 11:00:00'; ``` #### Sync Run 3 If there are additional updates in the source data: Now cursor field UpdatedAt value is 2024-04-21 10:00:00 | Name | Plan | Updated At | | ---------------- | ---- | ------------------- | | Charles Beaumont | paid | 2024-04-22 08:00:00 | | Eleanor Villiers | pro | 2024-04-22 09:00:00 | During the third sync run with the same cursor field, only the records for Charles Beaumont and Eleanor Villiers with 'Updated At' timestamp after the last sync would be fetched, continuing the process of minimal data transfer. ##### Query ```sql SELECT * FROM source_table WHERE updated_at >= '2024-04-21 10:00:00 '; ``` ### Handling Ambiguity and Inclusive Cursors When syncing data incrementally, we ensure at least one delivery. Limited cursor field granularity may cause sources to resend previously sent data. For example, if a cursor only tracks dates, distinguishing new from old data on the same day becomes unclear. #### Scenario Imagine sales transactions with a cursor field `transaction_date`. If we sync on April 1st and later sync on the same day, distinguishing new transactions becomes ambiguous. To mitigate this, we guarantee at least one delivery, allowing sources to resend data as needed. ### Known Limitations Modifications to underlying records without updating the cursor field may result in updated records not being picked up by the Incremental sync as expected. Edit or remove of cursor field can mess up tracking data changes, causing issues and data loss. So Don't change or remove the cursor field to keep sync smooth and reliable. # Full Refresh Full Refresh syncs replace existing data with new data. ### Overview The Full Refresh mode in AI Squared is a straightforward method used to sync data to a destination. It retrieves all available information from the source, regardless of whether it has been synced before. This mode is ideal for scenarios where you want to completely replace existing data in the destination with fresh data from the source. In the Full Refresh mode, new syncs will replace all existing data in the destination table with the new data from the source. This ensures that the destination contains the most up-to-date information available from the source. ### Example Behavior Consider the following scenario where we have a database table named `Users` in the destination: #### Before Sync | **id** | **name** | **email** | | ------ | -------- | --------------------------------------------- | | 1 | Alice | [alice@example.com](mailto:alice@example.com) | | 2 | Bob | [bob@example.com](mailto:bob@example.com) | #### New Data in Source | **id** | **name** | **email** | | ------ | -------- | --------------------------------------------- | | 1 | Alice | [alice@example.com](mailto:alice@example.com) | | 3 | Carol | [carol@example.com](mailto:carol@example.com) | | 4 | Dave | [dave@example.com](mailto:dave@example.com) | #### After Sync | **id** | **name** | **email** | | ------ | -------- | --------------------------------------------- | | 1 | Alice | [alice@example.com](mailto:alice@example.com) | | 3 | Carol | [carol@example.com](mailto:carol@example.com) | | 4 | Dave | [dave@example.com](mailto:dave@example.com) | In this example, notice how the previous user "Bob" is no longer present in the destination after the sync, and new users "Carol" and "Dave" have been added. # Incremental Incremental sync only transfer new or updated data, minimizing data transfer ### Overview Incremental syncing involves transferring only new or updated data, thus avoiding duplication of previously replicated data. This is achieved through deduplication using a unique primary key specified in the model. For initial syncs, it functions like a full refresh since all data is considered new. ### Example ### Initial State Suppose the following records already exist in our source: | Name | Plan | Updated At | | ---------------- | -------- | ---------- | | Charles Beaumont | freemium | 6789 | | Eleanor Villiers | freemium | 6790 | ### First sync In this sync, the delta contains an updated record for Charles: | Name | Plan | Updated At | | ---------------- | -------- | ---------- | | Charles Beaumont | freemium | 6791 | After this incremental sync, the data in the warehouse would now be: | Name | Plan | Updated At | | ---------------- | -------- | ---------- | | Charles Beaumont | freemium | 6791 | | Eleanor Villiers | freemium | 6790 | ### Second sync Let's assume in the next delta both customers have upgraded to a paid plan: | Name | Plan | Updated At | | ---------------- | ---- | ---------- | | Charles Beaumont | paid | 6795 | | Eleanor Villiers | paid | 6795 | The final data at the destination after this update will be: | Name | Plan | Updated At | | ---------------- | ---- | ---------- | | Charles Beaumont | paid | 6795 | | Eleanor Villiers | paid | 6795 | # Overview ### Introduction Syncs help in determining how the data appears in your destination. They are used to map the data from the source to the destination. Hero Light In order to create a sync, you need to have a source and a destination. The source is the data that you want to sync and the destination is where you want to sync the data to. ### Types of Syncs There are two types of syncs: 1. **Full Refresh Syncs**: This sync type replaces all the data in the destination with the data from the source. This is useful when you want to replace all the data in the destination with the data from the source. 2. **Incremental Syncs**: This sync type only syncs the data that has changed since the last sync. This is useful when you want to sync only the data that has changed since the last sync. ### Important Concepts 1. **Streams**: Streams in AI Squared are referred to the destination APIs that you want to sync the data to. For example, If you want to Sync the data to Salesforce, then `Account`, `Contact`, `Opportunity` are the streams. # Facebook Custom Audiences ## Connect AI Squared to Facebook Custom Audiences This guide will walk you through configuring the Facebook Custom Audiences Connector in AI Squared to manage your custom audiences effectively. ### Prerequisites Before you begin, make sure you have the following: 1. **Get your [System User Token](https://developers.facebook.com/docs/marketing-api/system-users/overview) from Facebook Business Manager account:** * Log in to your Facebook Business Manager account. * Go to Business Settings > Users > System Users. * Click "Add" to create a new system user if needed. * After creating the system user, access its details. * Generate a system user token by clicking "Generate New Token." * Copy the token for later use in the authentication process. 2. **Access to a Facebook Business Manager account:** * If you don't have an account, create one at [business.facebook.com](https://business.facebook.com/) by following the sign-up instructions. 3. **Custom Audiences:** * Log in to your Facebook Business Manager account. * Navigate to the Audiences section under Business Tools. * Create new custom audiences or access existing ones. ### Steps ### Authentication Authentication is supported via two methods: System user token and Log in with Facebook account. 1. **System User Token:** * **[access\_token](https://developers.facebook.com/docs/marketing-api/system-users/create-retrieve-update)**: Obtain a system user token from your Facebook Business Manager account in step 1 of the prerequisites. * **[ad\_account\_id](https://www.facebook.com/business/help/1492627900875762)**: Paste the system user token into the provided authentication field in AI Squared. * **[audience\_id](https://developers.facebook.com/docs/marketing-api/reference/custom-audience/)**: Obtain the audience ID from step 3 of the prerequisites. 2. **Log in with Facebook Account** *Coming soon* ### Supported Sync Modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | # Google Ads # Amplitude # Databricks ## Connect AI Squared to Databricks This guide will help you configure the Databricks Connector in AI Squared to access and use your Databricks data. ### Prerequisites Before proceeding, ensure you have the necessary Host URL and API Token from Databricks. ## Step-by-Step Guide to Connect to Databricks ## Step 1: Navigate to Databricks Start by logging into your Databricks account and navigating to the Databricks workspace. 1. Sign in to your Databricks account at [Databricks Login](https://accounts.cloud.databricks.com/login). 2. Once logged in, you will be directed to the Databricks workspace dashboard. ## Step 2: Locate Databricks Host URL and API Token Once you're logged into Databricks, you'll find the necessary configuration details: 1. **Host URL:** * The Host URL is the first part of the URL when you log into your Databricks. It will look something like `https://.databricks.com`. 2. **API Token:** * Click on your user icon in the upper right corner and select "Settings" from the dropdown menu. * In the Settings page, navigate to the "Developer" tab. * Here, you can create a new Access Token by clicking on Manage then "Generate New Token." Give it a name and set the expiration duration. * Once the token is generated, copy it as it will be required for configuring the connector. **Note:** This token will only be shown once, so make sure to store it securely. ## Step 3: Test the Databricks Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to Databricks from the AI Squared platform to ensure a connection is made. By following these steps, you’ve successfully set up a Databricks destination connector in AI Squared. You can now efficiently transfer data to your Databricks endpoint for storage or further distribution within AI Squared. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | Follow these steps to configure and test your Databricks connector successfully. # Google Analytics # Mixpanel # HubSpot HubSpot is a customer platform with all the software, integrations, and resources you need to connect your marketing, sales, content management, and customer service. HubSpot's connected platform enables you to grow your business faster by focusing on what matters most: your customers. ## Hubspot Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements Before initiating the Hubspot connector setup, ensure you have an created an Hubspot developer account. This setup requires us to create an private app in Hubspot with [superuser admin access](https://knowledge.hubspot.com/user-management/hubspot-user-permissions-guide#super-admin). [Hubspot Developer Signup](https://app.hubspot.com/signup-v2/developers/step/join-hubspot?hubs_signup-url=developers.hubspot.com/get-started\&hubs_signup-cta=developers-getstarted-app&_ga=2.53325096.1868562849.1588606909-500942594.1573763828). ### Destination Setup As mentioned earlier, this setup requires us to create an [private app](https://developers.hubspot.com/docs/api/private-apps) in Hubspot with superuser admin access. HubSpot private applications facilitate interaction with your HubSpot account's data through the platform's APIs. Granular control over individual app permissions allows you to specify the data each app can access or modify. This process generates a unique access token for each app, ensuring secure authentication. For AI Squared Open Source, we hubspot Private App Access Token for api authentication. Within your HubSpot account, access the settings menu from the main navigation bar. Navigate through the left sidebar menu to Integrations > Private Apps. Click the Create Private App button. On the Basic Info tab, configure your app's details: * Name: Assign a descriptive name for your app. * Logo: Upload a square image to visually represent your app (optional). * Description: Provide a brief explanation of your app's functionality. Navigate to the Scopes tab and select the desired access level (Write) for each data element your app requires. Utilize the search bar to locate specific scopes. After configuration, click Create app in the top right corner. A dialog box will display your app's unique access token. Click Continue creating to proceed. Once created, you can leverage the access token to setup hubspot in AI Squared destination section. # Salesforce ## Salesforce Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements Before initiating the Salesforce connector setup, ensure you have an appropriate Salesforce edition. This setup requires either the Enterprise edition of Salesforce, the Professional Edition with an API access add-on, or the Developer edition. For further information on API access within Salesforce, please consult the [Salesforce documentation](https://developer.salesforce.com/docs/). If you need a Developer edition of Salesforce, you can register at [Salesforce Developer Signup](https://developer.salesforce.com/signup). ### Destination Setup For AI Squared Open Source, certain OAuth credentials are necessary for authentication. These credentials include: * Access Token * Refresh Token * Instance URL * Client ID * Client Secret Start by logging into your Salesforce account with admin rights. Look for a Setup option in the menu at the top-right corner of the screen and click on it. On the left side of the screen, you'll see a menu. Click on Apps, then App Manager. Find a button that says New Connected App at the top right and click it. You'll be taken to a page to set up your new app. Here, you need to fill in some basic info: the name you want for your app, its API name (a technical identifier), and your email address. Now, look for a section named API (Enable OAuth Settings) and check the box for Enable OAuth Settings. There’s a box for a Callback URL; type in [https://login.salesforce.com/](https://login.salesforce.com/) there. You also need to pick some permissions from a list called Selected OAuth Scopes. Choose these: Access and manage your data (api), Perform requests on your behalf at any time (refresh\_token, offline\_access), Provide access to your data via the Web (web), and then add them to your app settings. Click Save to keep your new app's settings. Go back to where all your apps are listed (under Apps > App Manager), find the app you just created, and click Manage next to it. On the next screen, click Edit. There’s an option for OAuth policies; under Permitted Users, choose All users may self-authorize. Save your changes. Head back to your app list, find your new app again, and this time click View. Once more, go to the API (Enable OAuth Settings) section. Click on Manage Consumer Details. You need to write down two things: the Consumer Key and Consumer Secret. These are important because you'll use them to connect Salesforce. First, open Salesforce in your preferred web browser. To get the code, open a new tab and type in a special web address (URL). You'll need to change **CONSUMER\_KEY** to the Consumer Key you noted earlier. Also, replace **INSTANCE\_URL** with your specific Salesforce instance name (for example, ours is multiwoven-dev in [https://multiwoven-dev.develop.lightning.force.com/](https://multiwoven-dev.develop.lightning.force.com/)). ``` https://INSTANCE_URL.salesforce.com/services/oauth2/authorize?response_type=code&client_id=CONSUMER_KEY&redirect_uri=https://login.salesforce.com/ ``` If you see any alerts asking for permission, go ahead and allow them. After that, the browser will take you to a new webpage. Pay attention to this new web address because it contains the code you need. Save the code available in the new URL as shown in the below example. ``` https://login.salesforce.com/services/oauth2/success?code=aPrx0jWjRo8KRXs42SX1Q7A5ckVpD9lSAvxdKnJUApCpikQQZf.YFm4bHNDUlgiG_PHwWQ%3D%3Dclient_id = "3MVG9pRzvMkjMb6kugcl2xWhaCVwiZPwg17wZSM42kf6HqY4jmw6ocKKoYYLz4ztHqM1vWxMbZB6sxQQU" ``` Now, you'll use a tool called curl to ask for more keys, known as tokens. You'll type a command into your computer that includes the special code you just got. Remember to replace **CODE** with your code, and also replace **CONSUMER\_KEY** and **CONSUMER\_SECRET** with the details you saved from when you set up the app in Salesforce. ``` curl -X POST https://INSTANCE_URL.salesforce.com/services/oauth2/token?code=CODE&grant_type=authorization_code&client_id=CONSUMER_KEY&client_secret=CONSUMER_SECRET&redirect_uri=https://login.salesforce.com/ ``` After you send this command, you'll get back a response that includes your access\_token and refresh\_token. These tokens are what you'll use to securely access Salesforce data. ``` { "access_token": "access_token", "refresh_token": "refresh_token", "signature": "signature", "scope": "scopes", "id_token": "id_token", "instance_url": "https://multiwoven-dev.develop.my.salesforce.com", "id": "id", "token_type": "Bearer", "issued_at": "1707415379555", "api_instance_url": "https://api.salesforce.com" } ``` This way, you’re essentially getting the necessary permissions and access to work with Salesforce data in more detail. | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | Yes | | Full refresh | Coming soon | | Stream | Supported (Yes/No/Coming soon) | | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | | [Account](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_account.htm) | Yes | # Zoho # null # Zendesk Zendesk is a customer service software and support ticketing system. Zendesk's connected platform enables you to improve customer relationships by providing seamless support and comprehensive customer insights. ## Zendesk Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements Before initiating the Zendesk connector setup, ensure you have an active Zendesk account with admin privileges. This setup requires you to use your Zendesk username and password for authentication. [Zendesk Developer Signup](https://www.zendesk.com/signup) ### Destination Setup As mentioned earlier, this setup requires your Zendesk username and password with admin access for authentication. For Multiwoven Open Source, we use Zendesk username and password for authentication. Log into your Zendesk Developer account and navigate to the Admin Center by clicking on the gear icon in the sidebar. Zendesk Admin Console Within the Admin Center, go to Channels > API. Ensure that the Password access is enabled by toggling the switch. Zendesk Auth Enablement Ensure you have your Zendesk username and password. The username is typically your email address associated with the Zendesk account. Once you have your credentials, you can use the username and password to set up Zendesk in the Multiwoven destination section. # MariaDB ## Connect AI Squared to MariaDB This guide will help you configure the MariaDB Connector in AI Squared to access and transfer data to your MariaDB database. ### Prerequisites Before proceeding, ensure you have the necessary host, port, username, password, and database name from your MariaDB server. ## Step-by-Step Guide to Connect to MariaDB ## Step 1: Navigate to MariaDB Console Start by logging into your MariaDB Management Console and navigating to the MariaDB service. 1. Sign in to your MariaDB account on your local server or through the MariaDB Enterprise interface. 2. In the MariaDB console, select the service you want to connect to. ## Step 2: Locate MariaDB Configuration Details Once you're in the MariaDB console, you'll find the necessary configuration details: 1. **Host and Port:** * For local servers, the host is typically `localhost` and the default port is `3306`. * For remote servers, check your server settings or consult with your database administrator to get the correct host and port. * Note down the host and port as they will be used to connect to your MariaDB service. 2. **Username and Password:** * In the MariaDB console, you can find or create a user with the necessary permissions to access the database. * Note down the username and password as they are required for the connection. 3. **Database Name:** * List the available databases using the command `SHOW DATABASES;` in the MariaDB console. * Choose the database you want to connect to and note down its name. ## Step 3: Configure MariaDB Connector in Your Application Now that you have gathered all the necessary details, enter the following information in your application: * **Host:** The host of your MariaDB service. * **Port:** The port number of your MariaDB service. * **Username:** Your MariaDB service username. * **Password:** The corresponding password for the username. * **Database:** The name of the database you want to connect to. ## Step 4: Test the MariaDB Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to MariaDB from your application to ensure everything is set up correctly. By following these steps, you’ve successfully set up an MariaDB destination connector in AI Squared. You can now efficiently transfer data to your MariaDB endpoint for storage or further distribution within AI Squared. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | This guide will help you seamlessly connect your AI Squared application to MariaDB, enabling you to leverage your database's full potential. # MicrosoftSQL Microsoft SQL Server (Structured Query Language) is a proprietary relational database management system developed by Microsoft. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network. ## Setting MS SQL Connector in AI Squared To integrate Microsoft SQL with AI Squared, you need to establish a destination connector. This connector will enable AI Squared to load data from various sources efficiently. Below are the steps to set up the MS SQL Destination connector in AI Squared: ### Step 1: Access AI Squared * Log in to your AI Squared account. * Navigate to the `destinations` section where you can manage your destinations. ### Step 2: Create a New destination Connector * Click on the `Add destination` button. * Select `Microsoft SQL` from the list of available destination types. ### Step 3: Configure Connection Settings You'll need to provide the following details to establish a connection between AI Squared and your MicrosoftSQL Database: `Host` The hostname or IP address of the server where your MicrosoftSQL database is hosted. `Port` The port number on which your MicrosoftSQL server is listening (default is 1433). `Database` The name of the database you want to connect to. `Schema` The schema within your MicrosoftSQL database you wish to access. The default schema for Microsoft SQL Server is dbo. `Username` The username used to access the database. `Password` The password associated with the username. Enter these details in the respective fields on the connector configuration page and press continue. ### Step 4: Test the Connection * Once you've entered the necessary information. The next step is automated **Test Connection** feature to ensure that AI Squared can successfully connect to your MicrosoftSQL database. * If the test is successful, you'll receive a confirmation message. If not, double-check your entered details for any errors. ### Step 5: Finalize the destination Connector Setup * Save the connector settings to establish the destination connection. ## Notes * The Azure SQL Database firewall is a security feature that protects customer data by blocking access to the SQL Database server by default. To allow access, users can configure firewall rules to specify which IP addresses are permitted to access the database. [https://learn.microsoft.com/en-us/azure/azure-sql/database/firewall-configure?view=azuresql](https://learn.microsoft.com/en-us/azure/azure-sql/database/firewall-configure?view=azuresql) * Your credentials must be able to: Add/update/delete rows in your sync's table. * Get the connection information you need to connect to the database in Azure SQL Database. You'll need the fully qualified server name or host name, database name, and login information for the connection setup. * Sign in to the Azure portal. * Navigate to the SQL Databases or SQL Managed Instances page. * On the Overview page, review the fully qualified server name next to Server name for the database in Azure SQL Database or the fully qualified server name (or IP address) next to Host for an Azure SQL Managed Instance or SQL Server on Azure VM. To copy the server name or host name, hover over it and select the Copy icon. * More info at [https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-content-reference-guide?view=azuresql](https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-content-reference-guide?view=azuresql) # Oracle ## Connect AI Squared to Oracle This guide will help you configure the Oracle Connector in AI Squared to access and transfer data to your Oracle database. ### Prerequisites Before proceeding, ensure you have the necessary host, port, SID or service name, username, and password from your Oracle database. ## Step-by-Step Guide to Connect to Oracle database ### Step 1: Locate Oracle database Configuration Details In your Oracle database, you'll need to find the necessary configuration details: 1. **Host and Port:** * For local servers, the host is typically `localhost` and the default port is `1521`. * For remote servers, check your server settings or consult with your database administrator to get the correct host and port. * Note down the host and port as they will be used to connect to your Oracle database. 2. **SID or Service Name:** * To find your SID or Service name: 1. **Using SQL\*Plus or SQL Developer:** * Connect to your Oracle database using SQL\*Plus or SQL Developer. * Execute the following query: ```sql select instance from v$thread ``` or ```sql SELECT sys_context('userenv', 'service_name') AS service_name FROM dual; ``` * The result will display the SID or service name of your Oracle database. 2. **Checking the TNSNAMES.ORA File:** * Locate and open the `tnsnames.ora` file on your system. This file is usually found in the `ORACLE_HOME/network/admin` directory. * Look for the entry corresponding to your database connection. The `SERVICE_NAME` or `SID` will be listed within this entry. * Note down the SID or service name as it will be used to connect to your Oracle database. 3. **Username and Password:** * In the Oracle, you can find or create a user with the necessary permissions to access the database. * Note down the username and password as it will be used to connect to your Oracle database. ### Step 2: Configure Oracle Connector in Your Application Now that you have gathered all the necessary details, enter the following information in your application: * **Host:** The host of your Oracle database. * **Port:** The port number of your Oracle database. * **SID:** The SID or service name you want to connect to. * **Username:** Your Oracle username. * **Password:** The corresponding password for the username. ### Step 3: Test the Oracle Database Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to Oracle database from your application to ensure everything is set up correctly. By following these steps, you’ve successfully set up an Oracle database destination connector in AI Squared. You can now efficiently transfer data to your Oracle database for storage or further distribution within AI Squared. ## Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | This guide will help you seamlessly connect your AI Squared application to Oracle Database, enabling you to leverage your database's full potential. # PostgreSQL PostgreSQL popularly known as Postgres, is a powerful, open-source object-relational database system that uses and extends the SQL language combined with many features that safely store and scale data workloads. ## Setting Up a destination Connector in AI Squared To integrate PostgreSQL with AI Squared, you need to establish a destination connector. This connector will enable AI Squared to extract data from your PostgreSQL database efficiently. Below are the steps to set up the destination connector in AI Squared: ### Step 1: Access AI Squared * Log in to your AI Squared account. * Navigate to the `destinations` section where you can manage your data destinations. ### Step 2: Create a New destination Connector * Click on the `Add destination` button. * Select `PostgreSQL` from the list of available destination types. ### Step 3: Configure Connection Settings You'll need to provide the following details to establish a connection between AI Squared and your PostgreSQL database: `Host` The hostname or IP address of the server where your PostgreSQL database is hosted. `Port` The port number on which your PostgreSQL server is listening (default is 5432). `Database` The name of the database you want to connect to. `Schema` The schema within your PostgreSQL database you wish to access. `Username` The username used to access the database. `Password` The password associated with the username. Enter these details in the respective fields on the connector configuration page and press continue. ### Step 4: Test the Connection * Once you've entered the necessary information. The next step is automated **Test Connection** feature to ensure that AI Squared can successfully connect to your PostgreSQL database. * If the test is successful, you'll receive a confirmation message. If not, double-check your entered details for any errors. ### Step 5: Finalize the destination Connector Setup * Save the connector settings to establish the destination connection. ### Conclusion By following these steps, you've successfully set up a PostgreSQL destination connector in AI Squared. # null # null # SFTP Learn how to set up a SFTP destination connector in AI Squared to efficiently transfer data to your SFTP server. ## Introduction The Secure File Transfer Protocol (SFTP) is a secure method for transferring files between systems. Integrating SFTP with AI Squared allows you to efficiently transfer data to your SFTP server for storage or further distribution. This guide outlines the steps to set up an SFTP destination connector in AI Squared. ### Step 1: Access AI Squared 1. Log in to your AI Squared account. 2. Navigate to the **Destinations** section to manage your data destinations. ### Step 2: Create a New Destination Connector 1. Click on the **Add Destination** button. 2. Select **SFTP** from the list of available destination types. ### Step 3: Configure Connection Settings Provide the following details to establish a connection between AI Squared and your SFTP server: * **Host**: The hostname or IP address of the SFTP server. * **Port**: The port number used for SFTP connections (default is 22). * **Username**: Your username for accessing the SFTP server. * **Password**: The password associated with the username. * **Destination Path**: The directory path on the SFTP server where you want to store the files. * **Filename**: The name of the file to be uploaded to the SFTP server, appended with the current timestamp. Enter these details in the respective fields on the connector configuration page and press **Finish**. ### Step 4: Test the Connection 1. After entering the necessary information, use the automated **Test Connection** feature to ensure AI Squared can successfully connect to your SFTP server. 2. If the test is successful, you'll receive a confirmation message. If not, double-check your entered details for any errors. ### Step 5: Finalize the Destination Connector Setup 1. After a successful connection test, save the connector settings to establish the destination connection. ## Conclusion By following these steps, you've successfully set up an SFTP destination connector in AI Squared. You can now efficiently transfer data to your SFTP server for storage or further distribution within AI Squared. # HTTP Learn how to set up a HTTP destination connector in AI Squared to efficiently transfer data to your HTTP destination. ## Introduction The Hyper Text Transfer Protocol (HTTP) connector is a method of transerring data over the internet to specific url endpoints. Integrating the HTTP Destination connector with AI Squared allows you to efficiently transfer your data to HTTP endpoints of your choosing. This guide outlines how to setup your HTTP destination connector in AI Squared. ### Destination Setup Log in to your AI Squared account and navigate to the **Destinations** section to manage your data destinations. Click on the **Add Destination** button. Select **HTTP** from the list of available destination types. Provide the following details to establish a connection between AI Squared and your HTTP endpoint: * **Destination Url**: The HTTP address of where you are sending your data. * **Headers**: A list of key value pairs of your choosing. This can include any headers that are required to send along with your HTTP request. Enter these details in the respective fields on the connector configuration page and press **Finish**. After entering the necessary information, use the automated **Test Connection** feature to ensure AI Squared can successfully connect to your HTTP endpoint. If the test is successful, you'll receive a confirmation message. If not, double-check your entered details for any errors. After a successful connection test, save the connector settings to establish the destination connection. By following these steps, you've successfully set up an HTTP destination connector in AI Squared. You can now efficiently transfer data to your HTTP endpoint for storage or further distribution within AI Squared. | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | Yes | | Full refresh | No | # Braze # CleverTap # Iterable ## Connect AI Squared to Iterable This guide will help you configure the Iterable Connector in AI Squared to access and use your Iterable data. ### Prerequisites Before proceeding, ensure you have the necessary API Key from Iterable. ## Step-by-Step Guide to Connect to Iterable ## Step 1: Navigate to Iterable Start by logging into your Iterable account and navigating to the Iterable service. 1. Sign in to your Iterable account at [Iterable Login](https://www.iterable.com/login/). 2. Once logged in, you will be directed to the Iterable dashboard. ## Step 2: Locate Iterable API Key Once you're logged into Iterable, you'll find the necessary configuration details: 1. **API Key:** * Click on "Integrations" and select "API Keys" from the dropdown menu. * Here, you can create a new API key or use an existing one. Click on "+ New API key" if needed, and give it a name. * Once the API key is created, copy it as it will be required for configuring the connector. ## Step 3: Test the Iterable Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to Iterable from the AI Squared platform to ensure a connection is made. By following these steps, you’ve successfully set up an Iterable destination connector in AI Squared. You can now efficiently transfer data to your Iterable endpoint for storage or further distribution within AI Squared. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | Follow these steps to configure and test your Iterable connector successfully. # Klaviyo # Destination/Klaviyo ### Overview Enhance Your ECommerce Email Marketing Campaigns Using Warehouse Data in Klaviyo ### Setup 1. Create a [Klaviyo account](https://www.klaviyo.com/) 2. Generate a[ Private API Key](https://help.klaviyo.com/hc/en-us/articles/115005062267-How-to-Manage-Your-Account-s-API-Keys#your-private-api-keys3) and Ensure All Relevant Scopes are Included for the Streams You Wish to Replicate. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | Yes | | Full refresh | Coming soon | ### Supported streams | Stream | Supported (Yes/No/Coming soon) | | ---------------------------------------------------------------------------------- | ------------------------------ | | [Profiles](https://developers.klaviyo.com/en/v2023-02-22/reference/get_profiles) | Yes | | [Campaigns](https://developers.klaviyo.com/en/v2023-06-15/reference/get_campaigns) | Coming soon | | [Events](https://developers.klaviyo.com/en/reference/get_events) | Coming soon | | [Lists](https://developers.klaviyo.com/en/reference/get_lists) | Coming soon | # Stripe ## Overview Integrating customer data with subscription metrics from Stripe provides valuable insights into the actions that most frequently convert free accounts into paying ones. It also helps identify accounts that may be at risk of churning due to low activity levels. By recognizing these trends, you can proactively engage at-risk customers to prevent churn and enhance customer retention. ## Stripe Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements To authenticate the Stripe connector using AI Squared, you'll need a Stripe API key. While you can use an existing key, it's better to create a new restricted key specifically for AI Squared. Make sure to grant it write privileges only. Additionally, it's advisable to enable write privileges for all possible permissions and tailor the specific data you wish to synchronize within AI Squared. ### Set up Stripe Sign into your Stripe account. Click 'Developers' on the top navigation bar. At the top-left, click 'API keys'. Select '+ Create restricted key'. Name your key, and ensure 'Write' is selected for all permissions. Click 'Create key'. You may need to verify by entering a code sent to your email. | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | Yes | | Full refresh | Coming soon | | Stream | Supported (Yes/No/Coming soon) | | -------- | ------------------------------ | | Customer | Yes | | Product | Yes | # Airtable # Destination/Airtable ### Overview Airtable combines the simplicity of a spreadsheet with the complexity of a database. This cloud-based platform enables users to organize work, manage projects, and automate workflows in a customizable and collaborative environment. ### Prerequisite Requirements Ensure you have created an Airtable account before you begin. Sign up [here](https://airtable.com/signup) if you haven't already. ### Setup 1. **Generate a Personal Access Token** Start by generating a personal access token. Follow the guide [here](https://airtable.com/developers/web/guides/personal-access-tokens) for instructions. 2. **Grant Required Scopes** Assign the following scopes to your token for the necessary permissions: * `data.records:read` * `data.records:write` * `schema.bases:read` * `schema.bases:write` # Google Sheets - Service Account Google Sheets serves as an effective reverse ETL destination, enabling real-time data synchronization from data warehouses to a collaborative, user-friendly spreadsheet environment. It democratizes data access, allowing stakeholders to analyze, share, and act on insights without specialized skills. The platform supports automation and customization, enhancing decision-making and operational efficiency. Google Sheets transforms complex data into actionable intelligence, fostering a data-driven culture across organizations. Google Sheets is equipped with specific data capacity constraints, which, when exceeded, can lead to synchronization issues. Here's a concise overview of these limitations: * **Cell Limit**: A Google Sheets document is capped at `10 million` cells, which can be spread across one or multiple sheets. Once this limit is reached, no additional data can be added, either in the form of new rows or columns. * **Character Limit per Cell**: Each cell in Google Sheets can contain up to `50,000` characters. It's crucial to consider this when syncing data that includes fields with lengthy text. * **Column Limit**: A single worksheet within Google Sheets is limited to `18,278` columns. * **Worksheet Limit**: There is a cap of `200` worksheets within a single Google Sheets spreadsheet. Given these restrictions, Google Sheets is recommended primarily for smaller, non-critical data engagements. It may not be the optimal choice for handling expansive data operations due to its potential for sync failures upon reaching these imposed limits. ## Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements Before initiating the Google Sheet connector setup, ensure you have an created or access an [Google cloud account](https://console.cloud.google.com). ### Destination Setup * Navigate to the [Service Accounts](https://console.cloud.google.com/projectselector2/iam-admin/serviceaccounts) page in your Google Cloud console. * Choose an existing project or create a new one. * Click + Create service account, enter its name and description, then click Create and Continue. * Assign appropriate permissions, recommending the Editor role, then click Continue. * Access the [API Console > Credentials](https://console.cloud.google.com/apis/credentials) page, select your service account's email. * In the Keys tab, click + Add key and select Create new key. * Choose JSON as the Key type to download your authentication JSON key file. Click Continue. * Navigate to the [API Console > Library](https://console.cloud.google.com/apis/library) page. * Verify that the correct project is selected at the top. * Find and select the Google Sheets API. * Click ENABLE. * If your spreadsheet is link-accessible, no extra steps are needed. * If not, [grant your service account](https://support.google.com/a/answer/60781?hl=en\&sjid=11618327295115173982-AP) access to your spreadsheet. * Each worksheet becomes a separate source-connector stream in AI Squared. * Data is coerced to string format; nested structures need further processing for analysis. * AI Squared replicates text via Grid Sheets only; charts and images aren't supported. # Microsoft Excel ## Connect AI Squared to Microsoft Excel This guide will help you configure the Iterable Connector in AI Squared to access and use your Iterable data. ### Prerequisites Before proceeding, ensure you have the necessary Access Token from Microsoft Graph. ## Step-by-Step Guide to Connect to Microsoft Excel ## Step 1: Navigate to Microsoft Graph Explorer Start by logging into Microsoft Graph Explorer using your Microsoft account and consent to the required permissions. 1. Sign into Microsoft Graph Explorer at [developer.microsoft.com](https://developer.microsoft.com/en-us/graph/graph-explorer). 2. Once logged in, consent to the following under each category: * **Excel:** * worksheets in a workbook * used range in worksheet * **OneDrive:** * list items in my drive * **User:** * me 3. Once the following is consented to click Access token and copy the token ## Step 2: Navigate to Microsoft Excel Once you're logged into Microsoft Excel do the following: 1. **Create a new workbook:** * Create a new workbook in excel to have the data stored. * Once you have create the workbook open it and navigate to the sheet of you choosing. * In the sheet of your choosing make a table with the required headers you want to transfer data to. ## Step 3: Configure Microsoft Excel Connector in Your Application Now that you have gathered all the necessary details, enter the following information in your application: * **Token:** The access token from Microsoft Graph Explorer. ## Step 4: Test the Microsoft Excel Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to Microsoft Excel from the AI Squared platform to ensure a connection is made. By following these steps, you’ve successfully set up an Microsoft Excel destination connector in AI Squared. You can now efficiently transfer data to your Microsoft Excel workbook for storage or further distribution within AI Squared. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | Follow these steps to configure and test your Iterable connector successfully. # Salesforce Consumer Goods Cloud ## Overview Salesforce Consumer Goods Cloud is a specialized CRM platform designed to help companies in the consumer goods industry manage their operations more efficiently. It provides tools to optimize route-to-market strategies, increase sales performance, and enhance field execution. This cloud-based solution leverages Salesforce's robust capabilities to deliver data-driven insights, streamline inventory and order management, and foster closer relationships with retailers and customers. ### Key Features: * **Retail Execution**: Manage store visits, ensure product availability, and optimize shelf placement. * **Sales Planning and Operations**: Create and manage sales plans that align with company goals. * **Trade Promotion Management**: Plan, execute, and analyze promotional activities to maximize ROI. * **Field Team Management**: Enable field reps with tools and data to improve productivity and effectiveness. ## Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements When setting up an integration between Salesforce Consumer Goods Cloud and Multiwoven, certain credentials are required to authenticate and establish a secure connection. Below is a brief description of each credential needed: * **Username**: The Salesforce username used to log in. * **Password**: The password associated with the Salesforce username. * **Host**: The URL of your Salesforce instance (e.g., [https://login.salesforce.com](https://login.salesforce.com)). * **Security Token**: An additional security key that is appended to your password for API access from untrusted networks. * **Client ID** and **Client Secret**: These are part of the OAuth credentials required for authenticating an application with Salesforce. They are obtained when you set up a new "Connected App" in Salesforce for integrating with external applications. You may refer our [Salesforce CRM docs](https://docs.multiwoven.com/destinations/crm/salesforce#destination-setup) for further details. ### Setting Up Security Token in Salesforce Log in to your Salesforce account. Navigate to Settings or My Settings by first clicking on My Profile and then clicking **Settings** under the Personal Information section. Once inside the Settings page click on the Quick Find box and type "Reset My Security Token" to quickly navigate to the option. Click on Reset My Security Token under the Personal section. Salesforce will send the new security token to the email address associated with your account. If you do not see the option to reset the security token, it may be because your organization uses Single Sign-On (SSO) or has IP restrictions that negate the need for a token. | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | Yes | | Full refresh | Coming soon | | Stream | Supported (Yes/No/Coming soon) | | ----------- | ------------------------------ | | Account | Yes | | User | Yes | | Visit | Yes | | RetailStore | Yes | | RecordType | Yes | # null # Slack ## Usecase Notify sales or customer support teams about significant customer events, like contract renewals or support tickets, directly in Slack. Share real-time insights and reports in Slack channels to foster collaborative analysis and decision-making among teams. This is particularly useful for remote and distributed teams Integrate Slack with operational systems to streamline operations. For instance, sending real-time alerts about system downtimes, performance bottlenecks, or successful deployments to relevant engineering or operations Slack channels. Trigger marketing actions based on customer behavior. For example, if a customer action indicates high engagement, a notification can be sent to the marketing team to follow up with personalized content or offers. ## Slack Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements To access Slack through AI Squared, you must authenticate using an API Token. This authentication can be obtained through a Slack App. However, if you already possess one, it remains valid for use with this integration. Given that AI Squared operates as a reverse ETL platform, requiring write access to perform its functions, we recommend creating a restricted API key that permits write access specifically for AI Squared's use. This strategy enables you to maintain control over the extent of actions AI Squared can execute within your Slack environment, ensuring security and compliance with your data governance policies. Link to view your [Slack Apps](https://api.slack.com/apps). ### Destination Setup To facilitate the integration of your Slack destination connector with AI Squared, please follow the detailed steps below: Initiate the process by selecting the "Create New App" option. You will be required to create a Bot app from the ground up. To do this, select the "from scratch" option. Proceed by entering your desired App Name and selecting a workspace where the app will be deployed. Navigate to the **Add features and functionality** menu and select **Bots** to add this capability to your app. Within the menu on the side labeled as **Features** column, locate and click on **OAuth & Permissions**. In the "OAuth & Permissions" section, add the scope **chat:write** to define the permissions for your app. To finalize the Bot installation, click on "Install to workspace" found in the "OAuth Tokens for Your Workspace" section. Upon successful installation, a Bot User OAuth Token will be generated. It is crucial to copy this token as it is required for the configuration of the Slack destination connector within AI Squared. Additionally, acquiring the Channel ID is essential for configuring the Slack destination. This ID can be retrieved by right-clicking on the channel intended for message dispatch through the newly created bot. From the context menu, select **View channel details** Locate and copy the Channel ID, which is displayed at the lower left corner of the tab. # S3 ## Connect AI Squared to S3 This page describes how to add AWS S3 as a source. AI Squared lets you pull data from CSV and Parquet files stored in an Amazon S3 bucket and push them to downstream destinations. To get started, you need an S3 bucket and AWS credentials. ## Connector Configuration and Credentials Guide ### Prerequisites Before proceeding, ensure you have the necessary information based on how you plan to authenticate to AWS. The two types of authentication we support are: * IAM User with access id and secret access key. * IAM Role with ARN configured with an external ID so that AI Square can connect to your S3 bucket. Additional info you will need regardless of authentication type will be: * Region * Bucket name * The type of file we are working with (CSV or Parquet) * Path to the CSV or Parquet files ### Setting Up AWS Requirements Log in to your AWS account at [AWS Management Console](https://aws.amazon.com/console/). Navigate to the the **Users**. This can be found in the left navigation under "Access Management" -> "Users". Once inside the Users page, Select the User you would like to authenticate with. If there are no users to select, create one and make sure to give it the required permissions to read from S3 buckets. If you haven't created an access key pair before, click on "Create access key" to generate a new one. Make sure to copy the Secret Access Key as they are shown only once. After selecting the user, go to **Security Credentials** tab and in it you should be able to see the Access keys for that user. Log in to your AWS account at [AWS Management Console](https://aws.amazon.com/console/). The ARN is going to need an external ID which will be required during the configuration of the S3 source connector. The external ID will allow us to reach out to you S3 buckets and read data from it. You can generate an external Id using this [GUID generator tool](https://guidgenerator.com/). [Learn more about AWS STS external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html). Navigate to the the **Roles**. This can be found in the left navigation under "Access Management" -> "Roles". Select an existing role to edit or create a new one by clicking on "Create Role". The "Permissions Policy" should look something like this: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::{your-bucket-name}", "arn:aws:s3:::{your-bucket-name}/*" ] } ] } ``` The "Trust Relationship" should look something like this: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "Statement1", "Effect": "Allow", "Principal": { "AWS": "{iam-user-principal-arn}" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "{generated-external-id}" } } } ] } ``` ### Step 2: Locate AWS S3 Configuration Details Now you should be in the AWS and have found your credentials. Now we will navigate to the S3 service to find the necessary configuration details: 1. **IAM User Access Key and Secret Access Key or IAM Role ARN and External ID:** * This has been gathered from the previous step. 2. **Bucket:** * Once inside of the AWS S3 console you should be able to see the list of buckets available, if not go ahead and create a bucket by clicking on the "Create bucket" button. 3. **Region:** * In the same list showing the buckets, there's a region assotiated with it. 4. **Path:** * The path where the file you wish to read from. This field is optional and can be left blank. 5. **File type:** * The files within the path that was selected should help determine the file type. ### Step 3: Configure S3 Connector in Your Application Now that you have gathered all the necessary details enter the following information: * **Region:** The AWS region where your S3 bucket resources are located. * **Access Key ID:** Your AWS IAM user's Access Key ID. * **Secret Access Key:** The corresponding Secret Access Key. * **Bucket:** The name of the bucket you want to use. * **Path:** The path directory where the files are located at. * **File type:** The type of file (csv, parquet). ### Step 4: Test the S3 Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to S3 from your application to ensure everything is set up correctly. 3. Run a test query or check the connection status to verify successful connectivity. Your S3 connector is now configured and ready to query data from your S3 data catalog. ## Building a Model Query The S3 source connector is powered by [DuckDB S3 api support](https://duckdb.org/docs/extensions/httpfs/s3api.html). This allows us to use SQL queries to describe and/or fetch data from an S3 bucket, for example: ``` SELECT * FROM 's3://my-bucket/path/to/file/file.parquet'; ``` From the example, we can notice some details that are required in order to perform the query: * **FROM command: `'s3://my-bucket/path/to/file/file.parquet'`** You need to provide a value in the same format as the example. * **Bucket: `my-bucket`** In that format you will need to provide the bucket name. The bucket name needs to be the same one provided when configuring the S3 source connector. * **Path: `/path/to/file`** In that format you will need to provide the path to the file. The path needs to be the same one provided when configuring the S3 source connector. * **File name and type: `file.parquet`** In that format you will need to provide the file name and type at the end of the path. The file type needs to be the same one provided when configuring the S3 source connector. ## Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | YES | # AWS Athena ## Connect AI Squared to AWS Athena This guide will help you configure the AWS Athena Connector in AI Squared to access and use your AWS Athena data. ### Prerequisites Before proceeding, ensure you have the necessary access key, secret access key, region, workgroup, catalog, and output location from AWS Athena. ## Step-by-Step Guide to Connect to AWS Athena ## Step 1: Navigate to AWS Athena Console Start by logging into your AWS Management Console and navigating to the AWS Athena service. 1. Sign in to your AWS account at [AWS Management Console](https://aws.amazon.com/console/). 2. In the AWS services search bar, type "Athena" and select it from the dropdown. ## Step 2: Locate AWS Athena Configuration Details Once you're in the AWS Athena console, you'll find the necessary configuration details: 1. **Access Key and Secret Access Key:** * Click on your username at the top right corner of the AWS Management Console. * Choose "Security Credentials" from the dropdown menu. * In the "Access keys" section, you can create or view your access keys. * If you haven't created an access key pair before, click on "Create access key" to generate a new one. Make sure to copy the Access Key ID and Secret Access Key as they are shown only once. 2. **Region:** * The AWS region can be selected from the top right corner of the AWS Management Console. Choose the region where your AWS Athena resources are located or where you want to perform queries. 3. **Workgroup:** * In the AWS Athena console, navigate to the "Workgroups" section in the left sidebar. * Here, you can view the existing workgroups or create a new one if needed. Note down the name of the workgroup you want to use. 4. **Catalog and Database:** * Go to the "Data Source" section in the in the left sidebar. * Select the catalog that contains the databases and tables you want to query. Note down the name of the catalog and database. 5. **Output Location:** * In the AWS Athena console, click on "Settings". * Under "Query result location," you can see the default output location for query results. You can also set a custom output location if needed. Note down the output location URL. ## Step 3: Configure AWS Athena Connector in Your Application Now that you have gathered all the necessary details enter the following information: * **Access Key ID:** Your AWS IAM user's Access Key ID. * **Secret Access Key:** The corresponding Secret Access Key. * **Region:** The AWS region where your Athena resources are located. * **Workgroup:** The name of the workgroup you want to use. * **Catalog:** The name of the catalog containing your data. * **Schema:** The name of the database containing your data. * **Output Location:** The URL of the output location for query results. ## Step 4: Test the AWS Athena Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to AWS Athena from your application to ensure everything is set up correctly. 3. Run a test query or check the connection status to verify successful connectivity. Your AWS Athena connector is now configured and ready to query data from your AWS Athena data catalog. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | # AWS Sagemaker Model ## Connect AI Squared to AWS Sagemaker Model This guide will help you configure the AWS Sagemaker Model Connector in AI Squared to access your AWS Sagemaker Model Endpoint. ### Prerequisites Before proceeding, ensure you have the necessary access key, secret access key, and region from AWS. ## Step-by-Step Guide to Connect to an AWS Sagemaker Model Endpoint ## Step 1: Navigate to AWS Console Start by logging into your AWS Management Console. 1. Sign in to your AWS account at [AWS Management Console](https://aws.amazon.com/console/). ## Step 2: Locate AWS Configuration Details Once you're in the AWS console, you'll find the necessary configuration details: 1. **Access Key and Secret Access Key:** * Click on your username at the top right corner of the AWS Management Console. * Choose "Security Credentials" from the dropdown menu. * In the "Access keys" section, you can create or view your access keys. * If you haven't created an access key pair before, click on "Create access key" to generate a new one. Make sure to copy the Access Key ID and Secret Access Key as they are shown only once. 2. **Region:** * The AWS region can be selected from the top right corner of the AWS Management Console. Choose the region where your AWS Sagemaker resources is located and note down the region. ## Step 3: Configure AWS Sagemaker Model Connector in Your Application Now that you have gathered all the necessary details enter the following information: * **Access Key ID:** Your AWS IAM user's Access Key ID. * **Secret Access Key:** The corresponding Secret Access Key. * **Region:** The AWS region where your Sagemaker resources are located. # Google Big Query ## Connect AI Squared to BigQuery This guide will help you configure the BigQuery Connector in AI Squared to access and use your BigQuery data. ### Prerequisites Before you begin, you'll need to: 1. **Enable BigQuery API and Locate Dataset(s):** * Log in to the [Google Developers Console](https://console.cloud.google.com/apis/dashboard). * If you don't have a project, create one. * Enable the [BigQuery API for your project](https://console.cloud.google.com/flows/enableapi?apiid=bigquery&_ga=2.71379221.724057513.1673650275-1611021579.1664923822&_gac=1.213641504.1673650813.EAIaIQobChMIt9GagtPF_AIVkgB9Ch331QRREAAYASAAEgJfrfD_BwE). * Copy your Project ID. * Find the Project ID and Dataset ID of your BigQuery datasets. You can find this by querying the `INFORMATION_SCHEMA.SCHEMATA` view or by visiting the Google Cloud web console. 2. **Create a Service Account:** * Follow the instructions in our [Google Cloud Provider (GCP) documentation](https://cloud.google.com/iam/docs/service-accounts-create) to create a service account. 3. **Grant Access:** * In the Google Cloud web console, navigate to the [IAM](https://console.cloud.google.com/iam-admin/iam?supportedpurview=project,folder,organizationId) & Admin section and select IAM. * Find your service account and click on edit. * Go to the "Assign Roles" tab and click "Add another role". * Search and select the "BigQuery User" and "BigQuery Data Viewer" roles. * Click "Save". 4. **Download JSON Key File:** * In the Google Cloud web console, navigate to the [IAM](https://console.cloud.google.com/iam-admin/iam?supportedpurview=project,folder,organizationId) & Admin section and select IAM. * Find your service account and click on it. * Go to the "Keys" tab and click "Add Key". * Select "Create new key" and choose JSON format. * Click "Download". ### Steps ### Authentication Authentication is supported via the following: * **Dataset ID and JSON Key File** * **[Dataset ID](https://cloud.google.com/bigquery/docs/datasets):** The ID of the dataset within Google BigQuery that you want to access. This can be found in Step 1. * **[JSON Key File](https://cloud.google.com/iam/docs/keys-create-delete):** The JSON key file containing the authentication credentials for your service account. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | # ClickHouse ## Connect AI Squared to ClickHouse This guide will help you configure the ClickHouse Connector in AI Squared to access and use your ClickHouse data. ### Prerequisites Before proceeding, ensure you have the necessary URL, username, and password from ClickHouse. ## Step-by-Step Guide to Connect to ClickHouse ## Step 1: Navigate to ClickHouse Console Start by logging into your ClickHouse Management Console and navigating to the ClickHouse service. 1. Sign in to your ClickHouse account at [ClickHouse](https://clickhouse.com/). 2. In the ClickHouse console, select the service you want to connect to. ## Step 2: Locate ClickHouse Configuration Details Once you're in the ClickHouse console, you'll find the necessary configuration details: 1. **HTTP Interface URL:** * Click on the "Connect" button in your ClickHouse service. * In "Connect with" select HTTPS. * Find the HTTP interface URL, which typically looks like `http://:8443`. Note down this URL as it will be used to connect to your ClickHouse service. 2. **Username and Password:** * Click on the "Connect" button in your ClickHouse service. * Here, you will see the credentials needed to connect, including the username and password. * Note down the username and password as they are required for the HTTP connection. ## Step 3: Configure ClickHouse Connector in Your Application Now that you have gathered all the necessary details, enter the following information in your application: * **HTTP Interface URL:** The URL of your ClickHouse service HTTP interface. * **Username:** Your ClickHouse service username. * **Password:** The corresponding password for the username. ## Step 4: Test the ClickHouse Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to ClickHouse from your application to ensure everything is set up correctly. 3. Run a test query or check the connection status to verify successful connectivity. Your ClickHouse connector is now configured and ready to query data from your ClickHouse service. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | # Databricks ### Overview AI Squared enables you to transfer data from Databricks to various destinations by using Open Database Connectivity (ODBC). This guide explains how to obtain your Databricks cluster's ODBC URL and connect to AI Squared using your credentials. Follow the instructions to efficiently link your Databricks data with downstream platforms. ### Setup In your Databricks account, navigate to the "Workspaces" page, choose the desired workspace, and click Open workspace In your workspace, go the SQL warehouses and click on the relevant warehouse Go to the Connection details section.This tab shows your cluster's Server Hostname, Port, and HTTP Path, essential for connecting to AI Squared Then click on the create a personal token link to generate the personal access token ### Configuration | Field | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | **Server Hostname** | Visit the Databricks web console, locate your cluster, click for Advanced options, and go to the JDBC/ODBC tab to find your server hostname. | | **Port** | The default port is 443, although it might vary. | | **HTTP Path** | For the HTTP Path, repeat the steps for Server Hostname and Port. | | **Catalog** | Database catalog | | **Schema** | The initial schema to use when connecting. | # Databricks Model ### Overview AI Squared enables you to transfer data from a Databricks Model to various destinations or data apps. This guide explains how to obtain your Databricks Model URL and connect to AI Squared using your credentials. ### Setup Go to the Serving tab in Databricks, select the endpoint you want to configure, and locate the Databricks host and endpoint as shown below. Generate a personal access token by following the steps in the [Databricks documentation](https://docs.databricks.com/en/dev-tools/auth/pat.html). ### Configuration | Field | Description | | -------------------- | ---------------------------------------------- | | **databricks\_host** | The databricks-instance url | | **token** | Bearer token to connect with Databricks Model. | | **endpoint** | Name of the serving endpoint | # Google Vertex Model ## Connect AI Squared to Google Vertex Model This guide will help you configure the Google Vertex Model Connector in AI Squared to access your Google Vertex Model Endpoint. ### Prerequisites Before proceeding, ensure you have the necessary project id, endpoint id, region, and credential json from Google Vertex. ## Step-by-Step Guide to Connect to an Google Vertex Model Endpoint ## Step 1: Navigate to Google Cloud Console Start by logging into your Google Cloud Console. 1. Sign in to your google cloud account at [Google Cloud Console](https://console.cloud.google.com/). ## Step 2: Enable Vertex API * If you don't have a project, create one. * Enable the [Vertex API for your project](https://console.cloud.google.com/apis/library/aiplatform.googleapis.com). ## Step 3: Locate Google Vertex Configuration Details 1. **Project ID, Endpoint ID, and Region:** * In the search bar search and select "Vertex AI". * Choose "Online prediction" from the menu on the left hand side. * Select the region where your endpoint is and select your endpoint. Note down the Region that is shown. * Click on "SAMPLE REQUEST" and note down the Endpoint ID and Project ID 2. **JSON Key File:** * In the search bar search and select "APIs & Services". * Choose "Credentials" from the menu on the left hand side. * In the "Credentials" section, you can create or select your service account. * After selecting your service account goto the "KEYS" tab and click "ADD KEY". For Key type select JSON. ## Step 3: Configure Google Vertex Model Connector in Your Application Now that you have gathered all the necessary details enter the following information: * **Project ID:** Your Google Vertex Project ID. * **Endpoint ID:** Your Google Vertex Region ID. * **Region:** The Endpoint region where your Google Vertex resources are located. * **JSON Key File:** The JSON key file containing the authentication credentials for your service account. # HTTP Model Source Connector Guide on how to configure the HTTP Model Connector on the AI Squared platform ## Connect AI Squared to HTTP Model This guide will help you configure the HTTP Model Connector in AI Squared to access your HTTP Model Endpoint. ### Prerequisites Before starting, ensure you have the URL of your HTTP Model and any required headers for authentication or request configuration. ## Step-by-Step Guide to Connect to an HTTP Model Endpoint ## Step 1: Log in to AI Squared Sign in to your AI Squared account and navigate to the **Source** section. ## Step 2: Add a New HTTP Model Source Connector From AI/ML Sources in Sources click **Add Source** and select **HTTP Model** from the list of available source types. Configure HTTP Destination ## Step 3: Configure HTTP Connection Details Enter the following information to set up your HTTP connection: Configure HTTP Destination * **URL**: The URL where your model resides. * **Headers**: Any required headers as key-value pairs, such as authentication tokens or content types. * **Timeout**: The maximum time, in seconds, to wait for a response from the server before the request is canceled ## Step 4: Test the Connection Use the **Test Connection** feature to ensure that AI Squared can connect to your HTTP Model endpoint. If the test is successful, you’ll receive a confirmation message. If not, review your connection details. Configure HTTP Destination ## Step 5: Save the Connector Settings Once the connection test is successful, save the connector settings to establish the destination. # MariaDB ## Connect AI Squared to MariaDB This guide will help you configure the MariaDB Connector in AI Squared to access and use your MariaDB data. ### Prerequisites Before proceeding, ensure you have the necessary host, port, username, password, and database name from your MariaDB server. ## Step-by-Step Guide to Connect to MariaDB ## Step 1: Navigate to MariaDB Console Start by logging into your MariaDB Management Console and navigating to the MariaDB service. 1. Sign in to your MariaDB account on your local server or through the MariaDB Enterprise interface. 2. In the MariaDB console, select the service you want to connect to. ## Step 2: Locate MariaDB Configuration Details Once you're in the MariaDB console, you'll find the necessary configuration details: 1. **Host and Port:** * For local servers, the host is typically `localhost` and the default port is `3306`. * For remote servers, check your server settings or consult with your database administrator to get the correct host and port. * Note down the host and port as they will be used to connect to your MariaDB service. 2. **Username and Password:** * In the MariaDB console, you can find or create a user with the necessary permissions to access the database. * Note down the username and password as they are required for the connection. 3. **Database Name:** * List the available databases using the command `SHOW DATABASES;` in the MariaDB console. * Choose the database you want to connect to and note down its name. ## Step 3: Configure MariaDB Connector in Your Application Now that you have gathered all the necessary details, enter the following information in your application: * **Host:** The host of your MariaDB service. * **Port:** The port number of your MariaDB service. * **Username:** Your MariaDB service username. * **Password:** The corresponding password for the username. * **Database:** The name of the database you want to connect to. ## Step 4: Test the MariaDB Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to MariaDB from your application to ensure everything is set up correctly. 3. Run a test query or check the connection status to verify successful connectivity. Your MariaDB connector is now configured and ready to query data from your MariaDB service. ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | This guide will help you seamlessly connect your AI Squared application to MariaDB, enabling you to leverage your database's full potential. # Oracle ## Connect AI Squared to Oracle This guide will help you configure the Oracle Connector in AI Squared to access and transfer data to your Oracle database. ### Prerequisites Before proceeding, ensure you have the necessary host, port, SID or service name, username, and password from your Oracle database. ## Step-by-Step Guide to Connect to Oracle database ### Step 1: Locate Oracle database Configuration Details In your Oracle database, you'll need to find the necessary configuration details: 1. **Host and Port:** * For local servers, the host is typically `localhost` and the default port is `1521`. * For remote servers, check your server settings or consult with your database administrator to get the correct host and port. * Note down the host and port as they will be used to connect to your Oracle database. 2. **SID or Service Name:** * To find your SID or Service name: 1. **Using SQL\*Plus or SQL Developer:** * Connect to your Oracle database using SQL\*Plus or SQL Developer. * Execute the following query: ```sql select instance from v$thread ``` or ```sql SELECT sys_context('userenv', 'service_name') AS service_name FROM dual; ``` * The result will display the SID or service name of your Oracle database. 2. **Checking the TNSNAMES.ORA File:** * Locate and open the `tnsnames.ora` file on your system. This file is usually found in the `ORACLE_HOME/network/admin` directory. * Look for the entry corresponding to your database connection. The `SERVICE_NAME` or `SID` will be listed within this entry. * Note down the SID or service name as it will be used to connect to your Oracle database. 3. **Username and Password:** * In the Oracle, you can find or create a user with the necessary permissions to access the database. * Note down the username and password as it will be used to connect to your Oracle database. ### Step 2: Configure Oracle Connector in Your Application Now that you have gathered all the necessary details, enter the following information in your application: * **Host:** The host of your Oracle database. * **Port:** The port number of your Oracle database. * **SID:** The SID or service name you want to connect to. * **Username:** Your Oracle username. * **Password:** The corresponding password for the username. ### Step 3: Test the Oracle Database Connection After configuring the connector in your application: 1. Save the configuration settings. 2. Test the connection to Oracle database from your application to ensure everything is set up correctly. 3. Run a test query or check the connection status to verify successful connectivity. Your Oracle connector is now configured and ready to query data from your Oracle database. ## Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | This guide will help you seamlessly connect your AI Squared application to Oracle Database, enabling you to leverage your database's full potential. # PostgreSQL PostgreSQL popularly known as Postgres, is a powerful, open-source object-relational database system that uses and extends the SQL language combined with many features that safely store and scale data workloads. ## Setting Up a Source Connector in AI Squared To integrate PostgreSQL with AI Squared, you need to establish a source connector. This connector will enable AI Squared to extract data from your PostgreSQL database efficiently. Below are the steps to set up the source connector in AI Squared: ### Step 1: Access AI Squared * Log in to your AI Squared account. * Navigate to the `Sources` section where you can manage your data sources. ### Step 2: Create a New Source Connector * Click on the `Add Source` button. * Select `PostgreSQL` from the list of available source types. ### Step 3: Configure Connection Settings You'll need to provide the following details to establish a connection between AI Squared and your PostgreSQL database: `Host` The hostname or IP address of the server where your PostgreSQL database is hosted. `Port` The port number on which your PostgreSQL server is listening (default is 5432). `Database` The name of the database you want to connect to. `Schema` The schema within your PostgreSQL database you wish to access. `Username` The username used to access the database. `Password` The password associated with the username. Enter these details in the respective fields on the connector configuration page and press continue. ### Step 4: Test the Connection * Once you've entered the necessary information. The next step is automated **Test Connection** feature to ensure that AI Squared can successfully connect to your PostgreSQL database. * If the test is successful, you'll receive a confirmation message. If not, double-check your entered details for any errors. ### Step 5: Finalize the Source Connector Setup * Save the connector settings to establish the source connection. ### Conclusion By following these steps, you've successfully set up a PostgreSQL source connector in AI Squared. # Amazon Redshift ## Overview Amazon Redshift connector is built on top of JDBC and is based on the [Redshift JDBC driver](https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html). It allows you to connect to your Redshift data warehouse and extract data for further processing and analysis. ## Prerequisites Before proceeding, ensure you have the necessary Redshift credentials available, including the endpoint (host), port, database name, user, and password. You might also need appropriate permissions to create connections and execute queries within your Redshift cluster. ## Step-by-Step Guide to Connect Amazon Redshift ### Step 1: Navigate to the Sources Section Begin by accessing your AI Squared dashboard. From there: 1. Click on the Setup menu found on the sidebar. 2. Select the `Sources` section to proceed. ### Step 2: Add Redshift as a New Source Within the Sources section: 1. Find and click on the `Add Source` button. 2. From the list of data warehouse options, select **Amazon Redshift**. ### Step 3: Enter Redshift Credentials You will be prompted to enter the credentials for your Redshift cluster. This includes: **`Endpoint (Host)`** The URL of your Redshift cluster endpoint. **`Port`** The port number used by your Redshift cluster (default is 5439). **`Database Name`** The name of the database you wish to connect. **`User`** Your Redshift username. **`Password`** Your Redshift password. Make sure to enter these details accurately to ensure a successful connection. ### Step 4: Test the Connection Before finalizing the connection: Click on the `Test Connection` button. This step verifies that AI Squared can successfully connect to your Redshift cluster with the provided credentials. ### Step 5: Finalize Your Redshift Source Connection After a successful connection test: 1. Assign a name and a brief description to your Redshift source. This helps in identifying and managing your source within AI Squared. 2. Click `Save` to complete the setup process. ### Step 6: Configure Redshift User Permissions It is recommended to create a dedicated user with read-only access to the tables you want to query. Ensure that the new user has the necessary permissions to access the required tables and views. ```sql CREATE USER aisquared PASSWORD 'password'; GRANT USAGE ON SCHEMA public TO aisquared; GRANT SELECT ON ALL TABLES IN SCHEMA public TO aisquared; ``` Your Amazon Redshift data warehouse is now connected to AI Squared. You can now start creating models and running queries on your Redshift data. # Salesforce Consumer Goods Cloud ## Overview Salesforce Consumer Goods Cloud is a specialized CRM platform designed to help companies in the consumer goods industry manage their operations more efficiently. It provides tools to optimize route-to-market strategies, increase sales performance, and enhance field execution. This cloud-based solution leverages Salesforce's robust capabilities to deliver data-driven insights, streamline inventory and order management, and foster closer relationships with retailers and customers. ### Key Features: * **Retail Execution**: Manage store visits, ensure product availability, and optimize shelf placement. * **Sales Planning and Operations**: Create and manage sales plans that align with company goals. * **Trade Promotion Management**: Plan, execute, and analyze promotional activities to maximize ROI. * **Field Team Management**: Enable field reps with tools and data to improve productivity and effectiveness. ## Connector Configuration and Credential Retrieval Guide ### Prerequisite Requirements When setting up an integration between Salesforce Consumer Goods Cloud and Multiwoven, certain credentials are required to authenticate and establish a secure connection. Below is a brief description of each credential needed: * **Username**: The Salesforce username used to log in. * **Password**: The password associated with the Salesforce username. * **Host**: The URL of your Salesforce instance (e.g., [https://login.salesforce.com](https://login.salesforce.com)). * **Security Token**: An additional security key that is appended to your password for API access from untrusted networks. * **Client ID** and **Client Secret**: These are part of the OAuth credentials required for authenticating an application with Salesforce. They are obtained when you set up a new "Connected App" in Salesforce for integrating with external applications. You may refer our [Salesforce CRM docs](https://docs.multiwoven.com/destinations/crm/salesforce#destination-setup) for further details. ### Setting Up Security Token in Salesforce Log in to your Salesforce account. Navigate to Settings or My Settings by first clicking on My Profile and then clicking **Settings** under the Personal Information section. Once inside the Settings page click on the Quick Find box and type "Reset My Security Token" to quickly navigate to the option. Click on Reset My Security Token under the Personal section. Salesforce will send the new security token to the email address associated with your account. If you do not see the option to reset the security token, it may be because your organization uses Single Sign-On (SSO) or has IP restrictions that negate the need for a token. | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | Yes | | Full refresh | Coming soon | | Stream | Supported (Yes/No/Coming soon) | | ----------- | ------------------------------ | | Account | Yes | | User | Yes | | Visit | Yes | | RetailStore | Yes | | RecordType | Yes | # Snowflake # Source/Snowflake ### Overview This Snowflake source connector is built on top of the ODBC and is configured to rely on the Snowflake ODBC driver as described in Snowflake [documentation](https://docs.snowflake.com/en/developer-guide/odbc/odbc). ### Setup #### Authentication Authentication is supported via two methods: username/password and OAuth 2.0. 1. Login and Password | Field | Description | | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) | The host domain of the Snowflake instance. Must include the account, region, cloud environment, and end with snowflakecomputing.com. Example: accountname.us-east-2.aws.snowflakecomputing.com | | [Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses) | The Snowflake warehouse to be used for processing queries. | | [Database](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl) | The specific database in Snowflake to connect to. | | [Schema](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl) | The schema within the database you want to access. | | Username | The username associated with your account | | Password | The password associated with the username. | | [JDBC URL Params](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html) | (Optional) Additional properties to pass to the JDBC URL string when connecting to the database formatted as key=value pairs separated by the symbol &. Example: key1=value1\&key2=value2\&key3=value3 | 2. Oauth 2.0 Coming soon ### Supported sync modes | Mode | Supported (Yes/No/Coming soon) | | ---------------- | ------------------------------ | | Incremental sync | YES | | Full refresh | Coming soon | # Security and Compliance Common questions related to security, compliance, privacy policy and terms and conditions At AI Squared, we are dedicated to safeguarding your data and privacy. We adhere to industry best practices to ensure the security and protection of your information. We are SOC 2 Type II certified, demonstrating that we meet stringent standards for information security. This certification confirms that we have implemented robust policies and procedures to ensure the security, availability, processing integrity, and confidentiality of user data. You can trust that your data is safeguarded by the highest levels of security. ## Data Security We encrypt data at rest and in transit for all our customers. Using Azure's Key Vault, we securely manage encryption keys in accordance with industry best practices. Additionally, customer data is securely isolated from that of other customers, ensuring that your information remains protected and segregated at all times. ## Infrastructure Security We use Azure AKS to host our application, ensuring robust security through tools like Azure Key Vault, Azure Defender, and Azure Policy. We implement Role-Based Access Control (RBAC) to restrict access to customer data, ensuring that only authorized personnel have access. Your information is safeguarded by stringent security protocols, including limited access to our staff, and is protected by industry-leading infrastructure security measures. ## Reporting a Vulnerability If you discover a security issue in this project, please report it by sending an email to [security@squared.ai](mailto:security@squared.ai). We will respond to your report as soon as possible and will work with you to address the issue. We take security issues seriously and appreciate your help in making Multiwoven safe for everyone. # Workspace Management Learn how to create a new workspace, manage settings and workspace users. ## Introduction Workspaces enable the governance of data & AI activation. Each workspace within an organization's account will have self-contained data sources, data & AI models, syncs and business application destinations. ### Key workspace concepts * Organization: An AI Squared account that is a container for a set of workspaces. * Workspace: Represents a set of users and resources. One or more workspaces are contained within an organization. * User: An individual within a workspace, with a specific Role. A user can be a part of one or more workspaces. * Role: Defined by a set of rules that govern a user’s access to resources within a workspace * Resources: Product features within the workspace that enable the activation of data and AI. These include data sources, destinations, models, syncs, and more. ### Workspace settings You can access Workspace settings from within Settings on the left navigation menu. The workspace’s name and description can be edited at any time for clarity. ### Inviting users to a workspace You can view the list of active users on the Members tab, within Settings. Users can be invited or deleted from this screen. To invite a user, enter their email ID and choose their role. The invited user will receive an email invite (with a link that will expire after 30 days). The invite to a user can be cancelled or resent from this screen. ### Role-based access control (RBAC) Governance within workspaces is enabled by user Role-based access control (RBAC). * **Admins** have unrestricted access to all resources in the Organization’s account and all its workspaces. Admins can also create workspaces and manage the workspace itself, including inviting users and setting user roles. * **Members** belong to a single workspace, with access to all its resources. Members are typically part of a team or purpose that a workspace has been specifically set up for. * **Viewers** have read-only access to core resources within a workspace. Viewers can’t manage the workspace itself or add users. ### Creating a new workspace To create a workspace, use the drop-down on the left navigation panel that shows your current active workspace, click Manage Workspaces. Choose Create New Workspace. ### Moving between workspaces Your active workspace is visible on the left tab. The drop-down will allow you to view workspaces that you have access to, move between workspaces or create a workspace. # Overview AI Squared Enterprise SaaS is a `multi-tenant`, `cloud-native`, and `scalable` platform that enables organizations to sync data and AI model outputs from their data warehouse to business tools. It provides advanced capabilities like Reverse ETL, data transformation, and data governance for enterprise use cases. AI Squared Enterprise SaaS is built on top of the same technology stack as the AIS [open-source platform](https://docs.squared.ai/open-source/introduction). It provides a managed service with additional capabilities like multi-tenancy, advanced security, and enterprise-grade features. The high-level architecture of AI Squared Enterprise SaaS same as the architecture outlined in the [architecture section](https://docs.squared.ai/open-source/guides/architecture/introduction) of the open-source platform. ## Accessing AI Squared Enterprise SaaS ### Enterprise SaaS Cloud You can access the AI Squared Enterprise SaaS platform by visiting the [AI Squared Enterprise SaaS Cloud](https://cloud.squared.ai). You can sign up for a new account or log in with your existing account. ### Enterprise SaaS On-Premise AI Squared Enterprise SaaS is also available as an on-premise deployment. The on-premise deployment provides the same capabilities as the cloud but runs on your infrastructure. You can contact the AI Squared team to get more information about the on-premise deployment. # Self Hosting Enterprise Our self-hosted Enterprise SaaS offers 1 click deployment on your infrastructure. You can deploy and setup the platform on your private VPC across multiple cloud providers. The self-hosted Enterprise SaaS is container based and can be deployed on any Docker compatible infrastructure. ## Docker Compose To begin with, you can deploy the platform using Docker Compose using a simple virtual machine like EC2 or GCP VM. For more information on deploying the enterprise platform using Docker Compose, refer to the [**Docker Compose guide**](https://docs.squared.ai/open-source/guides/setup/docker-compose) in the open-source documentation. The Docker Compose guide provides detailed instructions on setting up the platform using Docker Compose. The difference between the open-source platform and the enterprise platform deployment is the docker image used for the platform. The enterprise platform uses the `squaredai/enterprise` docker image, which extends the platform with additional enterprise features. ## Kubernetes For a more scalable and production-grade deployment, you can deploy the platform on Kubernetes. The platform provides a [Helm chart](https://docs.squared.ai/open-source/guides/setup/helm) for deploying the platform on Kubernetes. You can refer to the [Environment Variables](https://docs.squared.ai/open-source/guides/setup/helm#environment-variables) section to configure the platform for your environment. You can also refer to the [Production Deployment](https://docs.squared.ai/open-source/guides/setup/docker-compose) section for a more detailed guide on deploying the platform to other cloud providers like AWS, GCP, and Azure. ## Accessing the Platform Once you have deployed the platform, you can access the platform by visiting `localhost:8000` to access the platform UI. You can sign up for a new account and log in to the platform, you should be able to see the platform dashboard. Hero Light In case you are facing any issues with the deployment, you can reach out to the AI Squared team for support. # Reverse ETL ### What if I Alter My Model? Changes in your model configuration impact the sync process, particularly for SaaS destinations. Only the mapped columns in your sync configurations are tracked for changes. ### What Happens When I Delete a Model? If you delete a model, all syncs associated with it are also deleted. However, the source and destination configurations remain intact. ### Can I Sync Data from Multiple Sources to a Single Destination? Yes, you can sync data from multiple sources to a single destination. You can create multiple syncs for each source and map them to the same destination. # Overview # null export function openSearch() { document.getElementById('search-bar-entry').click(); }
AI Squared Documentation

What can we help you build?

Choose a topic below or simply{' '} get started
Learn how to use AI Squared with our step-by-step guides. API reference, SDKs, and other developer tools for AI Squared. Explore AI Squared's open-source projects and deployments. Get help and find resources for AI Squared and related tools. Find solutions to common issues and errors in AI Squared. Stay up-to-date with the latest features and updates in AI Squared.
# Commit Message Guidelines Multiwoven follows the following format for all commit messages. Format: `([]) : ` ## Example ``` feat(CE): add source/snowflake connector ^--^ ^--^ ^------------^ | | | | | +-> Summary in present tense. | | | +-------> Edition: CE for Community Edition or EE for Enterprise Edition. | +-------------> Type: chore, docs, feat, fix, refactor, style, or test. ``` Supported Types: * `feat`: (new feature for the user, not a new feature for build script) * `fix`: (bug fix for the user, not a fix to a build script) * `docs`: (changes to the documentation) * `style`: (formatting, missing semi colons, etc; no production code change) * `refactor`: (refactoring production code, eg. renaming a variable) * `test`: (adding missing tests, refactoring tests; no production code change) * `chore`: (updating grunt tasks etc; no production code change) Sample messages: * feat(CE): add source/snowflake connector * feat(EE): add google sso References: * [https://gist.github.com/joshbuchea/6f47e86d2510bce28f8e7f42ae84c716](https://gist.github.com/joshbuchea/6f47e86d2510bce28f8e7f42ae84c716) * [https://www.conventionalcommits.org/](https://www.conventionalcommits.org/) * [https://seesparkbox.com/foundry/semantic\_commit\_messages](https://seesparkbox.com/foundry/semantic_commit_messages) * [http://karma-runner.github.io/1.0/dev/git-commit-msg.html](http://karma-runner.github.io/1.0/dev/git-commit-msg.html) # Contributor Code of Conduct Contributor Covenant Code of Conduct ## Our Pledge In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. ## Our Standards Examples of behavior that contributes to creating a positive environment include: * Using welcoming and inclusive language * Being respectful of differing viewpoints and experiences * Gracefully accepting constructive criticism * Focusing on what is best for the community * Showing empathy towards other community members Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances * Trolling, insulting/derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or electronic address, without explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting ## Our Responsibilities Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. Maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. ## Scope This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at \[your email]. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/) , version 1.4, available at [https://www.contributor-covenant.org/version/1/4/code-of-conduct.html]() For answers to common questions about this code of conduct, see [https://www.contributor-covenant.org/faq]() # Overview Hero Light The aim of our community to provide anyone with the assistance they need, connect them with fellow users, and encourage them to contribute to the growth of the Multiwoven ecosystem. ## Getting Help from the Community How to get help from the community? * Join our Slack channel and ask your question in the relevant channel. * Share as much information as possible about your issue, including screenshots, error messages, and steps to reproduce the issue. * If you’re reporting a bug, please include the steps to reproduce the issue, the expected behavior, and the actual behavior. ### Github Issues If you find a bug or have a feature request, please open an issue on GitHub. To open an issue for a specific repository, go to the repository and click on the `Issues` tab. Then click on the `New Issue` button. **Multiwoven server** issues can be reported [here](https://github.com/Multiwoven/multiwoven-server/issues). **Multiwoven frontend** issues can be reported [here](https://github.com/Multiwoven/multiwoven-ui/issues). **Multiwoven integration** issues can be reported [here](https://github.com/Multiwoven/multiwoven-integrations/issues). ### Contributing to Multiwoven We welcome contributions to the Multiwoven ecosystem. Please read our [contributing guidelines](https://github.com/Multiwoven/multiwoven/blob/main/CONTRIBUTING.md) to get started. We're always looking for ways to improve our documentation. If you find any mistakes or have suggestions for improvement, please [open an issue](https://github.com/Multiwoven/multiwoven/issues/new) on GitHub. # Release Process The release process at Multiwoven is fully automated through GitHub Actions. Here's an overview of our automation stages, each facilitated by specific GitHub Actions: * **Action**: [Release Workflow](https://github.com/Multiwoven/multiwoven/actions/workflows/release.yaml) * **Description**: Every Tuesday, a new release is automatically generated with a minor version tag (e.g., v0.4.0) following semantic versioning rules. This process also creates a pull request (PR) for release notes that summarize the changes in the new version. * **Additional Triggers**: The same workflow can be manually triggered to create a patch version (e.g., v0.4.1 for quick fixes) or a major version (e.g., v1.0.0 for significant architectural changes). This is done using the workflow dispatch feature in GitHub Actions. * **Action**: [Create Release Note on Merge](https://github.com/Multiwoven/multiwoven/actions/workflows/create-release-notes.yaml) * **Description**: When the release notes PR is merged, it triggers the creation of a new release with detailed [release notes](https://github.com/Multiwoven/multiwoven/releases/tag/v0.4.0) on GitHub. * **Description**: Docker images need to be manually released based on the newly created tags from the GitHub Actions. * **Actions**: * [Build and push Multiwoven server docker image to Docker Hub](https://github.com/Multiwoven/multiwoven/actions/workflows/server-docker-hub-push-tags.yaml): This action handles the server-side Docker image push to docker hub with tag as latest and the new release tag i.e **v0.4.0** * [Build and push Multiwoven UI docker image to Docker Hub](https://github.com/Multiwoven/multiwoven/actions/workflows/ui-docker-hub-push-tags.yaml): This action handles the user interface Docker image to docker hub with tag as latest and the new release tag i.e **v0.4.0** # Slack Code of Conduct ## Introduction At Multiwoven, we firmly believe that diversity and inclusion are the bedrock of a vibrant and effective community. We are committed to creating an environment that embraces a wide array of backgrounds and perspectives, and we want to clearly communicate our position on this. ## Our Commitment We aim to foster a community that is safe, supportive, and friendly for all members, regardless of their experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or any other defining characteristics. ## Scope These guidelines apply to all forms of behavior and communication within our community spaces, both online and offline, including one-on-one interactions. This extends to any behavior that could impact the safety and well-being of community members, regardless of where it occurs. ## Expected Behaviors * **Be Welcoming:** Create an environment that is inviting and open to all. * **Be Kind:** Treat others with respect, understanding, and compassion. * **Support Each Other:** Actively look out for the well-being of fellow community members. ## Multiwoven Slack Etiquette Guidelines To maintain a respectful, organized, and efficient communication environment within the Multiwoven community, we ask all members to adhere to the following etiquette guidelines on Slack: ## Etiquette Rules 1. **Be Respectful to Everyone:** Treat all community members with kindness and respect. A positive attitude fosters a collaborative and friendly environment. 2. **Mark Resolved Questions:** If your query is resolved, please indicate it by adding a ✅ reaction or a reply. This helps in identifying resolved issues and assists others with similar questions. 3. **Avoid Reposting Questions:** If your question remains unanswered after 24 hours, review it for clarity and revise if necessary. If you still require assistance, you may tag @navaneeth for further attention. 4. **Public Posts Over Direct Messages:** Please ask questions in public channels rather than through direct messages, unless you have explicit permission. Sharing questions and answers publicly benefits the entire community. 5. **Minimize Use of Tags:** Our community is active and responsive. Please refrain from over-tagging members. Reserve tagging for urgent matters to respect everyone's time and attention. 6. **Use Threads for Detailed Discussions:** To keep the main channel tidy, please use threads for ongoing discussions. This helps in keeping conversations organized and the main channel uncluttered. ## Conclusion Following these etiquette guidelines will help ensure that our Slack workspace remains a supportive, efficient, and welcoming space for all members of the Multiwoven community. Your cooperation is greatly appreciated! # Architecture Overview Multiwoven is structured into two primary components: the server and the connectors. The server delivers all the essential horizontal services needed for configuring and executing data movement tasks, such as the[ User Interface](https://github.com/Multiwoven/multiwoven-ui), [API](https://github.com/Multiwoven/multiwoven-server), Job Scheduling, etc., and is organised as a collection of microservices. Connectors are developed within the [multiwoven-integrations](https://github.com/Multiwoven/multiwoven-integrations) Ruby gem, which pushes and pulls data to and from various sources and destinations. These connectors are constructed following the [Multiwoven Protocol](https://docs.multiwoven.com/guides/architecture/multiwoven-protocol), which outlines the interface for transferring data between a source and a destination. 1. [Multiwoven-UI](https://github.com/Multiwoven/multiwoven-ui) - User interface to interact with [ multiwoven-server](https://github.com/Multiwoven/multiwoven-server). 2. [Multiwoven-Server](https://github.com/Multiwoven/multiwoven-server) - Multiwoven’s control plane. All operations in Multiwoven such as creating sources, destinations, connections, managing configurations, etc., are configured and invoked from the server. 3. Database: Stores all connector/sync information. 4. [Temporal ](https://temporal.io/)- Orchestrates the the sync workflows. 5. Multiwoven-Workers - The worker connects to a source connector, pulls the data, and writes it to a destination. The workers' code resides in the [ multiwoven-server](https://github.com/Multiwoven/multiwoven-server) repo. # Multiwoven Protocol ### Introduction Multiwoven [protocol](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L4) defines a set of interfaces for building connectors. Connectors can be implemented independent of our server application, this protocol allows developers to create connectors without requiring in-depth knowledge of our core platform. ### Concepts **[Source](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L66)** - A source in business data storage typically refers to data warehouses like Snowflake, AWS Redshift and Google BigQuery, as well as databases. **[Destination](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L66)** - A destination is a tool or third party service where source data is sent and utilised, often by end-users. It includes CRM systems, ad platforms, marketing automation, and support tools. **[Stream](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L105)** - A Stream defines the structure and metadata of a resource, such as a database table, REST API resource, or data stream, outlining how users can interact with it using query or request. ***Fields*** | Field | Description | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `name` | A string representing the name of the stream. | | `action` (optional) | Defines the action associated with the stream, e.g., "create", "update", or "delete". | | `json_schema` | A hash representing the JSON schema of the stream. | | `supported_sync_modes` (optional) | An array of supported synchronization modes for the stream. | | `source_defined_cursor` (optional) | A boolean indicating whether the source has defined a cursor for the stream. | | `default_cursor_field` (optional) | An array of strings representing the default cursor field(s) for the stream. | | `source_defined_primary_key` (optional) | An array of arrays of strings representing the source-defined primary key(s) for the stream. | | `namespace` (optional) | A string representing the namespace of the stream. | | `url` (optional) | A string representing the URL of the API stream. | | `request_method` (optional) | A string representing the request method (e.g., "GET", "POST") for the API stream. | | `batch_support` | A boolean indicating whether the stream supports batching. | | `batch_size` | An integer representing the batch size for the stream. | | `request_rate_limit` | An integer value, specifying the maximum number of requests that can be made to the user data API within a given time limit unit. | | `request_rate_limit_unit` | A string value indicating the unit of time for the rate limit. | | `request_rate_concurrency` | An integer value which limits the number of concurrent requests. | **[Catalog](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L123)** - A Catalog is a collection of Streams detailing the data within a data store represented by a Source/Destination eg: Catalog = Schema, Streams = List\[Tables] ***Fields*** | Field | Description | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `streams` | An array of Streams detailing the data within the data store. This encapsulates various data streams available for synchronization or processing, each potentially with its own schema, sync modes, and other configurations. | | `request_rate_limit` | An integer value, specifying the maximum number of requests that can be made to the user data API within a given time limit unit. This serves to prevent overloading the system by limiting the rate at which requests can be made. | | `request_rate_limit_unit` | A string value indicating the unit of time for the rate limit, such as "minute" or "second". This defines the time window in which the `request_rate_limit` applies. | | `request_rate_concurrency` | An integer value which limits the number of concurrent requests that can be made. This is used to control the load on the system by restricting how many requests can be processed at the same time. | | `schema_mode ` | A string value that identifies the schema handling mode for the connector. Supported values include **static, dynamic, and schemaless**. This parameter is crucial for determining how the connector handles data schema. | {" "} Rate limit specified in catalog will applied to stream if there is no stream specific rate limit is defined.{" "} **[Model](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L86)** - Models specify the data to be extracted from a source ***Fields*** * `name` (optional): A string representing the name of the model. * `query`: A string representing the query used to extract data from the source. * `query_type`: A type representing the type of query used by the model. * `primary_key`: A string representing the primary key of the model. **[Sync](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L134)** - A Sync sets the rules for data transfer from a chosen source to a destination ***Fields*** * `source`: The source connector from which data is transferred. * `destination`: The destination connector where data is transferred. * `model`: The model specifying the data to be transferred. * `stream`: The stream defining the structure and metadata of the data to be transferred. * `sync_mode`: The synchronization mode determining how data is transferred. * `cursor_field` (optional): The field used as a cursor for incremental data transfer. * `destination_sync_mode`: The synchronization mode at the destination. ### Interfaces The output of each method in the interface is encapsulated in an [MultiwovenMessage](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L170), serving as an envelope for the message's return value. These are omitted in interface explanations for sake of simplicity. #### Common 1. `connector_spec() -> ConnectorSpecification` Description - [connector\_spec](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/base_connector.rb#L10) returns information about how the connector can be configured Input - `None` Output - [ConnectorSpecification](https://github.com/Multiwoven/multiwoven-integrations/blob/6462867b1a2698b4c30ae5abcdf3219a207a28d9/lib/multiwoven/integrations/protocol/protocol.rb#L49) \-One of the main pieces of information the specification shares is what information is needed to configure an Actor. * **`documentation_url`**:\ URL providing information about the connector. * **`stream_type`**:\ The type of stream supported by the connector. Possible values include: * `static`: The connector catalog is static. * `dynamic`: The connector catalog is dynamic, which can be either schemaless or with a schema. * `user_defined`: The connector catalog is defined by the user. * **`connector_query_type`**:\ The type of query supported by the connector. Possible values include: * `raw_sql`: The connector is SQL-based. * `soql`: Specifically for Salesforce. * `ai_ml`: Specific for AI model source connectors. * **`connection_specification`**:\ The properties required to connect to the source or destination. * **`sync_mode`**:\ The synchronization modes supported by the connector. 2. `meta_data() -> Hash` Description - [meta\_data](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/base_connector.rb#L17) returns information about how the connector can be shown in the multiwoven ui eg: icon, labels etc. Input - `None` Output - `Hash`. Sample hash can be found [here](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/source/bigquery/config/meta.json) 3. `check_connection(connection_config) -> ConnectionStatus` Description: The [check\_connection](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/base_connector.rb#L21) method verifies if a given configuration allows successful connection and access to necessary resources for a source/destination, such as confirming Snowflake database connectivity with provided credentials. It returns a success response if successful or a failure response with an error message in case of issues like incorrect passwords Input - `Hash` Output - [ConnectionStatus](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L37) 4. `discover(connection_config) -> Catalog` Description: The [discover](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/base_connector.rb#L26) method identifies and outlines the data structure in a source/destination. Eg: Given a valid configuration for a Snowflake source, the discover method returns a list of accessible tables, formatted as streams. Input - `Hash` Output - [Catalog](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L121) #### Source [Source](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/source_connector.rb) implements the following interface methods including the common methods. ``` connector_spec() -> ConnectorSpecification meta_data() -> Hash check_connection(connection_config) -> ConnectionStatus discover(connection_config) -> Catalog read(SyncConfig) ->Array[RecordMessage] ``` 1. `read(SyncConfig) ->Array[RecordMessage]` Description -The [read](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/source_connector.rb#L6) method extracts data from a data store and outputs it as RecordMessages. Input - [SyncConfig](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L132) Output - List\[[RecordMessage](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L93)] #### Destination [Destination](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/destination_connector.rb) implements the following interface methods including the common methods. ``` connector_spec() -> ConnectorSpecification meta_data() -> Hash check_connection(connection_config) -> ConnectionStatus discover(connection_config) -> Catalog write(SyncConfig,Array[records]) -> TrackingMessage ``` 1. `write(SyncConfig,Array[records]) -> TrackingMessage` Description -The [write](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/core/destination_connector.rb#L6C11-L6C40) method loads data to destinations. Input - `Array[Record]` Output - [TrackingMessage](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb#L157) Note: Complete multiwoven protocol models can be found [here](https://github.com/Multiwoven/multiwoven-integrations/blob/main/lib/multiwoven/integrations/protocol/protocol.rb) ### Acknowledgements We've been significantly influenced by the [Airbyte protocol](https://github.com/airbytehq/airbyte-protocol), and their design choices greatly accelerated our project's development. # Sync States # Overview This document details the states and transitions of sync operations, organizing the sync process into specific statuses and run states. These categories are vital for managing data flow during sync operations, ensuring successful and efficient execution. ## Sync Status Definitions Each sync run operation can be in one of the following states, which represent the sync run's current status: | State | Description | | ------------ | ------------------------------------------------------------------------------------------------- | | **Healthy** | A state indicating the successful completion of a recent sync run operation without any issues. | | **Disabled** | Indicates that the sync operation has been manually turned off and will not run until re-enabled. | | **Pending** | Assigned immediately after a sync is set up, signaling that no sync runs have been initiated yet. | | **Failed** | Denotes a sync operation that encountered an error, preventing successful completion. | > **Note:** Ensure that sync configurations are regularly reviewed to prevent prolonged periods in the Disabled or Failed states. ### Sync State Transitions The following describes the allowed transitions between the sync states: * **Pending ➔ Healthy**: Occurs when a sync run completes successfully. * **Pending ➔ Failed**: Triggered if a sync run fails or is aborted. * **Failed ➔ Healthy**: A successful sync run after a previously failed attempt. * **Any state ➔ Disabled**: Reflects the manual disabling or enabling of the sync operation. ## Sync Run Status Definitions | Status | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Started** | Indicates that the sync operation has begun. This status serves as the initial state of a new sync run operation after being triggered. | | **Querying** | The sync is currently querying a source with its associated model to retrieve the latest data. This involves moving data to a temporary table called "SyncRecord". | | **Queued** | Indicates the sync is scheduled for execution, following the successful transfer of source data to the "SyncRecord" table. This marks the completion of the preparation phase, with the sync now ready to transmit data to the destination as per system scheduling and resource availability. | | **In Progress** | The sync is actively transferring data from the "SyncRecord" table to the destination. This phase marks the actual update or insertion of data into the destination database, reflecting the final step of the sync process. | | **Success** | The sync run is completed successfully without any issues. | | **Paused** | Indicates a temporary interruption occurred while transferring data from the "SyncRecord" table to the destination. The sync is paused but designed to automatically resume in a subsequent run, ensuring continuity of the sync process. | | **Aborted/Failed** | The sync has encountered an error that prevents it from completing successfully. | ### Sync Run State Transitions The following describes the allowed transitions between the sync run states: * **Started ➔ Querying**: Transition post-initiation as data retrieval begins. * **Querying ➔ Queued**: After staging data in the "SyncRecord" table, indicating readiness for transmission. * **Queued ➔ In Progress**: Commences as the sync operation begins writing data to the destination, based on availability of system resources. * **In Progress ➔ Success**: Marks the successful completion of data transmission. * **In Progress ➔ Paused**: Triggered by a temporary interruption in the sync process. * **Paused ➔ In Progress**: Signifies the resumption of a sync operation post-interruption. * **In Progress ➔ Aborted/Failed**: Initiated when an error prevents the successful completion of the sync operation. # Technical Stack ### Frameworks * **Ruby on Rails** * **Typescript** * **ReactJS** ## Database & Workers * **PostgreSQL** * **Temporal** * **Redis** ## Deployment * **Docker** * **Kubernetes** * **Helm** ## Monitoring * **Prometheus** * **Grafana** ## CI/CD * **Github Actions** ## Testing * **RSpec** * **Cypress** # Azure AKS (Kubernetes) ## Deploying Multiwoven on Azure Kubernetes Service (AKS) This guide will walk you through setting up Multiwoven on AKS. We'll cover configuring and deploying an AKS cluster after which, you can refer to the Helm Charts section of our guide to install Multiwoven into it. **Prerequisites** * An active Azure subscription * Basic knowledge of Kuberenetes and Helm **Note:** AKS clusters are not free. Please refer to [https://azure.microsoft.com/en-us/pricing/details/kubernetes-service/#pricing](https://azure.microsoft.com/en-us/pricing/details/kubernetes-service/#pricing) for current pricing information. **1. AKS Cluster Deployment:** 1. **Select a Resource Group for your deployment:** * Navigate to your Azure subscription and select a Resource Group or, if necessary, start by creating a new Resource Group. 2. **Initiate AKS Deployment** * Select the **Create +** button at the top of the overview section of your Resource Group which will take you to the Azure Marketplace. * In the Azure Marketplace, type **aks** into the search field at the top. Select **Azure Kuberenetes Service (AKS)** and create. 3. **Configure your AKS Cluster** * **Basics** * For **Cluster Preset Configuration**, we recommend **Dev/Test** for Development deployments. * For **Resource Group**, select your Resource Group. * For **AKS Pricing Tier**, we recommend **Standard**. * For **Kubernetes version**, we recommend sticking with the current **default**. * For **Authentication and Authorization**, we recommend **Local accounts with Kubernetes RBAC** for simplicity. * **Node Pools** * Leave defaults * **Networking** * For **Network Configuration**, we recommend the **Azure CNI** network configuration for simplicity. * For **Network Policy**, we recommend **Azure**. * **Integrations** * Leave defaults * **Monitoring** * Leave defaults, however, to reduce costs, you can uncheck **Managed Prometheus** which will automatically uncheck **Managed Grafana**. * **Advanced** * Leave defaults * **Tags** * Add tags if necessary, otherwise, leave defaults. * **Review + Create** * If there are validation errors that arise during the review, like a missed mandatory field, address the errors and create. If there are no validation errors, proceed to create. * Wait for your deployment to complete before proceeding. 4. **Connecting to your AKS Cluster** * In the **Overview** section of your AKS cluster, there is a **Connect** button at the top. Choose whichever method suits you best and follow the on-screen instructions. Make sure to run at least one of the test commands to verify that your kubectl commands are being run against your new AKS cluster. 5. **Deploying Multiwoven** * Please refer to the **Helm Charts** section of our guide to proceed with your installation of Multiwoven!\ [Helm Chart Deployment Guide](https://docs.squared.ai/open-source/guides/setup/helm) # Azure VMs ## Deploying Multiwoven on Azure VMs This guide will walk you through setting up Multiwoven on an Azure VM. We'll cover launching the VM, installing Docker, running Multiwoven with its dependencies, and finally, accessing the Multiwoven UI. **Prerequisites:** * An Azure account with an active VM (Ubuntu recommended). * Basic knowledge of Docker, Azure, and command-line tools. * Docker Compose installed on your local machine. **Note:** This guide uses environment variables for sensitive information. Replace the placeholders with your own values before proceeding. **1. Azure VM Setup:** 1. **Launch an Azure VM:** Choose an Ubuntu VM with suitable specifications for your workload. **Network Security Group Configuration:** * Open port 22 (SSH) for inbound traffic from your IP address. * Open port 8000 (Multiwoven UI) for inbound traffic from your IP address (optional). **SSH Key Pair:** Create a new key pair or use an existing one to connect to your VM. 2. **Connect to your VM:** Use SSH to connect to your Azure VM. **Example:** ``` ssh -i /path/to/your-key-pair.pem azureuser@ ``` Replace `/path/to/your-key-pair.pem` with the path to your key pair file and `` with your VM's public IP address. 3. **Update and upgrade:** Run `sudo apt update && sudo apt upgrade -y` to ensure your system is up-to-date. **2. Docker and Docker Compose Installation:** 1. **Install Docker:** Follow the official Docker installation instructions for Ubuntu: [https://docs.docker.com/engine/install/](https://docs.docker.com/engine/install/) 2. **Install Docker Compose:** Download the latest version from the Docker Compose releases page and place it in a suitable directory (e.g., `/usr/local/bin/docker-compose`). Make the file executable: `sudo chmod +x /usr/local/bin/docker-compose`. 3. **Start and enable Docker:** Run `sudo systemctl start docker` and `sudo systemctl enable docker` to start Docker and configure it to start automatically on boot. **3. Download Multiwoven `docker-compose.yml` file and Configure Environment:** 1. **Download the file:** ``` curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/docker-compose.yaml ``` 2. **Download the `.env` file:** ``` curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/.env.production ``` 3. Rename the file .env.production to .env and update the environment variables if required. ```bash mv .env.production .env ``` 4. \*\*Configure `.env`, This file holds environment variables for various services. Replace the placeholders with your own values, including: * `DB_PASSWORD` and `DB_USERNAME` for your PostgreSQL database * `REDIS_PASSWORD` for your Redis server * (Optional) Additional environment variables specific to your Multiwoven configuration **Example `.env` file:** ``` DB_PASSWORD=your_db_password DB_USERNAME=your_db_username REDIS_PASSWORD=your_redis_password # Modify your Multiwoven-specific environment variables here ``` **4. Run Multiwoven with Docker Compose:** 1. **Start Multiwoven:** Navigate to the `multiwoven` directory and run `docker-compose up -d`. This will start all Multiwoven services in the background, including the Multiwoven UI. **5. Accessing Multiwoven UI:** Open your web browser and navigate to `http://:8000` (replace `` with your VM's public IP address). You should now see the Multiwoven UI. **6. Stopping Multiwoven:** To stop Multiwoven, navigate to the `multiwoven` directory and run. ```bash docker-compose down ``` **7. Upgrading Multiwoven:** When a new version of Multiwoven is released, you can upgrade the Multiwoven using the following command. ```bash docker-compose pull && docker-compose up -d ``` Make sure to run the above command from the same directory where the `docker-compose.yml` file is present. **Additional Notes:** **Note**: the frontend and backend services run on port 8000 and 3001, respectively. Make sure you update the **VITE\_API\_HOST** environment variable in the **.env** file to the desired backend service URL running on port 3000. * Depending on your network security group configuration, you might need to open port 8000 (Multiwoven UI) for inbound traffic. * For production deployments, consider using a reverse proxy (e.g., Nginx) and a domain name with SSL/TLS certificates for secure access to the Multiwoven UI. # Docker Deploying Multiwoven using Docker Below steps will guide you through deploying Multiwoven on a server using Docker Compose. We require PostgreSQL database to store meta data for Multiwoven. We will use Docker Compose to deploy Multiwoven and PostgreSQL. Note: If you are setting up Multiwoven on your local machine, you can skip this section and refer to [Local Setup](/guides/setup/docker-compose-dev) section. ## Prerequisites * [Docker](https://docs.docker.com/get-docker/) * [Docker Compose](https://docs.docker.com/compose/install/) All our Docker images are available in x86\_64 architecture, make sure your server supports x86\_64 architecture. ## Deployment options Multiwoven can be deployed using two different options for PostgreSQL database. 1. Create a new directory for Multiwoven and navigate to it. ```bash mkdir multiwoven cd multiwoven ``` 2. Download the production `docker-compose.yml` file from the following link. ```bash curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/docker-compose.yaml ``` 3. Download the `.env.production` file from the following link. ```bash curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/.env.production ``` 4. Rename the file .env.production to .env and update the environment variables if required. ```bash mv .env.production .env ``` 5. Start the Multiwoven using the following command. ```bash docker-compose up -d ``` 6. Stopping Multiwoven To stop the Multiwoven, use the following command. ```bash docker-compose down ``` 7. Upgrading Multiwoven When a new version of Multiwoven is released, you can upgrade the Multiwoven using the following command. ```bash docker-compose pull && docker-compose up -d ``` Make sure to run the above command from the same directory where the `docker-compose.yml` file is present. 1. Create a new directory for Multiwoven and navigate to it. ```bash mkdir multiwoven cd multiwoven ``` 2. Download the production `docker-compose.yml` file from the following link. ```bash curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/docker-compose-cloud-postgres.yaml ``` 3. Rename the file .env.production to .env and update the **PostgreSQL** environment variables. `DB_HOST` - Database Host `DB_USERNAME` - Database Username `DB_PASSWORD` - Database Password The default port for PostgreSQL is 5432. If you are using a different port, update the `DB_PORT` environment variable. ```bash mv .env.production .env ``` 4. Start the Multiwoven using the following command. ```bash docker-compose up -d ``` ## Accessing Multiwoven Once the Multiwoven is up and running, you can access it using the following URL and port. Multiwoven Server URL: ```http http://:3000 ``` Multiwoven UI Service: ```http http://:8000 ``` If you are using a custom domain you can update the `API_HOST` and `UI_HOST` environment variable in the `.env` file. ### Important considerations * Make sure to update the environment variables in the `.env` file before starting the Multiwoven. * Make sure to take regular **backups** of the PostgreSQL database. To restore the backup, you can use the following command. ```bash cat dump.sql | docker exec -i --user postgres psql -U postgres ``` * If you are using a custom domain, make sure to update the `API_HOST` and `UI_HOST` environment variables in the `.env` file. # Docker **WARNING** The following guide is intended for developers to set-up Multiwoven locally. If you are a user, please refer to the [Self-Hosted](/guides/setup/docker-compose) guide. ## Prerequisites * [Docker](https://docs.docker.com/get-docker/) * [Docker Compose](https://docs.docker.com/compose/install/) **Note**: if you are using Mac or Windows, you will need to install [Docker Desktop](https://www.docker.com/products/docker-desktop) instead of just docker. Docker Desktop includes both docker and docker-compose. Verify that you have the correct versions installed: ```bash docker --version docker-compose --version ``` ## Installation 1. Clone the repository ```bash git clone git@github.com:Multiwoven/multiwoven.git ``` 2. Navigate to the `multiwoven` directory ```bash cd multiwoven ``` 3. Initialize .env file ```bash cp .env.example .env ``` **Note**: Refer to the [Environment Variables](/guides/setup/environment-variables) section for details on the ENV variables used in the Docker environment. 4. Build docker images ```bash docker-compose build ``` Note: The default build architecture is for **x86\_64**. If you are using **arm64** architecture, you will need to run the below command to build the images for arm64. ```bash TARGETARCH=arm64 docker-compose ``` 5. Start the containers ```bash docker-compose up ``` 6. Stop the containers ```bash docker-compose down ``` ## Usage Once the containers are running, you can access the `Multiwoven UI` at [http://localhost:8000](http://localhost:8000). The `multiwoven API` is available at [http://localhost:3000/api/v1](http://localhost:3000/api/v1). ## Running Tests 1. Running the complete test suite on the multiwoven server ```bash docker-compose exec multiwoven-server bundle exec rspec ``` ## Troubleshooting To cleanup all images and containers, run the following commands: ```bash docker rmi -f $(docker images -q) docker rm -f $(docker ps -a -q) ``` prune all unused images, containers, networks and volumes **Danger:** This will remove all unused images, containers, networks and volumes. ```bash docker system prune -a ``` Please open a new issue at [https://github.com/Multiwoven/multiwoven/issues](https://github.com/Multiwoven/multiwoven/issues) if you run into any issues or join our [Slack]() to chat with us. # Digital Ocean Droplets Coming soon... # Digital Ocean Kubernetes Coming soon... # AWS EC2 ## Deploying Multiwoven on AWS EC2 Using Docker Compose This guide walks you through setting up Multiwoven, on an AWS EC2 instance using Docker Compose. We'll cover launching the instance, installing Docker, running Multiwoven with its dependencies, and finally, accessing the Multiwoven UI. **Important Note:** at present, TLS is required. This means that to successfully deploy the Platform via docker-compose, you will need access to a DNS record set as well as the ability to obtain a valid TLS certificate from a Certificate Authority. You can obtain a free TLS certificates via tools like CertBot, Amazon Certificate Manager (if using an AWS Application Load Balancer to front an EC2 instance), letsencrypt-nginx-proxy-companion (if you add an nginx proxy to the docker-compose file to front the other services), etc. **Prerequisites:** * An active AWS account * Basic knowledge of AWS and Docker * A private repository access key (please contact your AIS point of contact if you have not received one) **Notes:** * This guide uses environment variables for sensitive information. Replace the placeholders with your own values before proceeding. * This guide uses an Application Load Balancer (ALB) to front the EC2 instance for ease of enabling secure TLS communication with the backend using an Amazon Certificate Manager (ACM) TLS certificate. These certificates are free of charge and ACM automatically rotates them every 90 days. While the ACM certificate is free, the ALB is not. You can refer to the following document for current ALB pricing: [ALB Pricing Page](https://aws.amazon.com/elasticloadbalancing/pricing/?nc=sn\&loc=3). **1. Obtain TLS Certificate (Requires access to DNS Record Set)** **1.1** In the AWS Management Console, navigate to Amazon Certificate Manager and request a new certificate. 1.2 Unless your organization has created a Private CA (Certificate Authority), we recommend requesting a public certificate. 1.3 Request a single ACM certificate that can verify all three of your chosen subdomains for this deployment. DNS validation is recommended for automatic rotation of your certificate but this method requires access to your domain's DNS record set. 1.4 Once you have added your selected sub-domains, scroll down and click **Request**. 5. Once your request has been made, you will be taken to a page that will describe your certificate request and its current status. Scroll down a bit and you will see a section labeled **Domains** with 3 subdomains and 1 CNAME validation record for each. These records need to be added to your DNS record set. Please refer to your organization's internal documentation or the documentation of your DNS service for further instruction on how to add DNS records to your domain's record set.
**Note:** For automatic certificate rotation, you need to leave these records in your record set. If they are removed, automatic rotation will fail. 6. Once your ACM certificate has been issued, note the ARN of your certificate and proceed. **2. Create and Configure Application Load Balancer and Target Groups** 1. In the AWS Management Console, navigate to the EC2 Dashboard and select **Load Balancers**. {" "} 2. On the next screen select **Create** under **Application Load Balancer**. {" "} 3. Under **Basic configuration** name your load balancer. If you are deploying this application within a private network, select **Internal**. Otherwise, select **Internet-facing**. Consult with your internal Networking team if you are unsure as this setting can not be changed post-deployment and you will need to create an entirely new load balancer to correct this. {" "} 4. Under **Network mapping**, select a VPC and write it down somewhere for later use. Also, select 2 subnets (2 are **required** for an Application Load Balancer) and write them down too for later use.
**Note:** If you are using the **internal** configuration, select only **private** subnets. If you are using the **internet-facing** configuration, you must select **public** subnets and they must have routes to an **Internet Gateway**. 5. Under **Security groups**, select the link to **create a new security group** and a new tab will open. 6. Under **Basic details**, name your security group and provide a description. Be sure to pick the same VPC that you selected for your load balancer configuration. 7. Under **Inbound rules**, create rules for HTTP and HTTPS and set the source for both rules to **Anywhere**. This will expose inbound ports 80 and 443 on the load balancer. Leave the default **Outbound rules** allowing for all outbound traffic for simplicity. Scroll down and select **Create security group**. 8. Once the security group has been created, close the security group tab and return to the load balancer tab. On the load balancer tab, in the **Security groups** section, hit the refresh icon and select your newly created security group. If the VPC's **default security group** gets appended automatically, be sure to remove it before proceeding. 9. Under **Listeners and routing** in the card for **Listener HTTP:80**, select **Create target group**. A new tab will open. 10. Under **Basic configuration**, select **Instances**. 11. Scroll down and name your target group. This first one will be for the Platform's web app so you should name it accordingly. Leave the protocol set to HTTP **but** change the port value to 8000. Also, make sure that the pre-selected VPC matches the VPC that you selected for the load balancer. Scroll down and click **Next**. 12. Leave all defaults on the next screen, scroll down and select **Create target group**. Repeat this process 2 more times, once for the **Platform API** on **port 3000** and again for **Temporal UI** on **port 8080**. You should now have 3 target groups. 13. Navigate back to the load balancer configuration screen and hit the refresh button in the card for **Listener HTTP:80**. Now, in the target group dropdown, you should see your 3 new target groups. For now, select any one of them. There will be some further configuration needed after the creation of the load balancer. 14. Now, click **Add listener**. 15. Change the protocol to HTTPS and in the target group dropdown, again, select any one of the target groups that you previously created. 16. Scroll down to the **Secure listener settings**. Under **Default SSL/TLS server certificate**, select **From ACM** and in the **Select a certificate** dropdown, select the certificate that you created in Step 1. In the dropdown, your certificate will only show the first subdomain that you listed when you created the certificate request. This is expected behavior. **Note:** If you do not see your certificate in the dropdown list, the most likely issues are:
(1) your certificate has not yet been successfully issued. Navigate back to ACM and verify that your certificate has a status of **Issued**. (2) you created your certificate in a different region and will need to either recreate your load balancer in the same region as your certificate OR recreate your certificate in the region in which you are creating your load balancer. 17. Scroll down to the bottom of the page and click **Create load balancer**. Load balancers take a while to create, approximately 10 minutes or more. However, while the load balancer is creating, copy the DNS name of the load balancer and create CNAME records in your DNS record set, pointing all 3 of your chosen subdomains to the DNS name of the load balancer. Until you complete this step, the deployment will not work as expected. You can proceed with the final steps of the deployment but you need to create those CNAME records. 18. At the bottom of the details page for your load balancer, you will see the section **Listeners and rules**. Click on the listener labeled **HTTP:80**. 19. Check the box next to the **Default** rule and click the **Actions** dropdown. 20. Scroll down to **Routing actions** and select **Redirect to URL**. Leave **URI parts** selected. In the **Protocol** dropdown, select **HTTPS** and set the port value to **443**. This configuration step will automatically redirect all insecure requests to the load balancer on port 80 (HTTP) to port 443 (HTTPS). Scroll to the bottom and click **Save**. 21. Return to the load balancer's configuration page (screenshot in step 18) and scroll back down to the *Listeners and rules* section. This time, click the listener labled **HTTPS:443**. 22. Click **Add rule**. 23. On the next page, you can optionally add a name to this new rule. Click **Next**. 24. On the next page, click **Add condition**. In the **Add condition** pop-up, select **Host header** from the dropdown. For the host header, put the subdomain that you selected for the Platform web app and click **Confirm** and then click **Next**. 25. One the next page, under **Actions**. Leave the **Routing actions** set to **Forward to target groups**. From the **Target group** dropdown, select the target group that you created for the web app. Click **Next**. 26. On the next page, you can set the **Priority** to 1 and click **Next**. 27. On the next page, click **Create**. 28. Repeat steps 24 - 27 for the **api** (priority 2) and **temporal ui** (priority 3). 29. Optionally, you can also edit the default rule so that it **Returns a fixed response**. The default **Response code** of 503 is fine. **3. Launch EC2 Instance** 1. Navigate to the EC2 Dashboard and click **Launch Instance**. 2. Name your instance and select **Ubuntu 22.04 or later** with **64-bit** architecture. 3. For instance type, we recommend **t3.large**. You can find EC2 on-demand pricing here: [EC2 Instance On-Demand Pricing](https://aws.amazon.com/ec2/pricing/on-demand). Also, create a **key pair** or select a pre-existing one as you will need it to SSH into the instance later. 4. Under **Network settings**, click **Edit**. 5. First, verify that the listed **VPC** is the same one that you selected for the load balancer. Also, verify that the pre-selected subnet is one of the two that you selected earlier for the load balancer as well. If either is incorrect, make the necessary changes. If you are using **private subnets** because your load balancer is **internal**, you do not need to auto-assign a public IP. However, if you chose **internet-facing**, you may need to associate a public IP address with your instance so you can SSH into it from your local machine. 6. Under **Firewall (security groups)**, we recommend that you name the security group but this is optional. After naming the security security group, click the button \*Add security group rule\*\* 3 times to create 3 additional rules. 7. In the first new rule (rule 2), set the port to **3000**. Click the **Source** input box and scroll down until you see the security group that you previously created for the load balancer. Doing this will firewall inbound traffic to port 3000 on the EC2 instance, only allowing inbound traffic from the load balancer that you created earlier. Do the same for rules 3 and 4, using ports 8000 and 8080 respectively. 8. Scroll to the bottom of the screen and click on **Advanced Details**. 9. In the **User data** box, paste the following to automate the installation of **Docker** and **docker-compose**. ``` Content-Type: multipart/mixed; boundary="//" MIME-Version: 1.0 --// Content-Type: text/cloud-config; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="cloud-config.txt" #cloud-config cloud_final_modules: - [scripts-user, always] --// Content-Type: text/x-shellscript; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="userdata.txt" #!/bin/bash sudo mkdir ais cd ais # install docker sudo apt-get update yes Y | sudo apt-get install apt-transport-https ca-certificates curl software-properties-common sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - echo | sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt-get update yes Y | sudo apt-get install docker-ce sudo systemctl status docker --no-pager && echo "Docker status checked" # install docker-compose sudo apt-get install -y jq VERSION=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r .tag_name) sudo curl -L "https://github.com/docker/compose/releases/download/${VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose docker-compose --version sudo systemctl enable docker ``` 10. In the right-hand panel, click **Launch instance**. **4. Register EC2 Instance in Target Groups** 1. Navigate back to the EC2 Dashboard and in the left panel, scroll down to **Target groups**. 2. Click on the name of the first listed target group. 3. Under **Registered targets**, click **Register targets**. 4. Under **Available instances**, you should see the instance that you just created. Check the tick-box next to the instance and click **Include as pending below**. Once the instance shows in **Review targets**, click **Register pending targets**. 5. **Repeat steps 2 - 4 for the remaining 2 target groups.** **5. Deploy AIS Platform** 1. SSH into the EC2 instance that you created earlier. For assistance, you can navigate to your EC2 instance in the EC2 dashboard and click the **Connect** button. In the **Connect to instance** screen, click on **SSH client** and follow the instructions on the screen. 2. Verify that **Docker** and **docker-compose** were successfully installed by running the following commands ``` sudo docker --version sudo docker-compose --version ``` You should see something similar to 3. Change directory to the **ais** directory and download the AIS Platform docker-compose file and the corresponding .env file. ``` cd \ais sudo curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/docker-compose.yaml sudo curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/.env.production && sudo mv /ais/.env.production /ais/.env ``` Verify the downloads ``` ls -a ``` You should see the following 4. You will need to edit both files a little before deploying. First open the .env file. ``` sudo nano .env ``` **There are 3 required changes.**

**(1)** Set the variable **VITE\_API\_HOST** so the UI knows to send requests to your **API subdomain**.

**(2)** If not present already, add a variable **Track** and set its value to **no**.

**(3)** If not present already, add a variable **ALLOWED\_HOST**. The value for this is dependent on how you selected your subdomains earlier. This variable only allows for a single step down in subdomain so if, for instance, you selected ***app.mydomain.com***, ***api.mydomain.com*** and ***temporal.mydomain.com*** you would set the value to **.mydomain.com**. If you selected ***app.c1.mydomain.com***, ***api.c1.mydomain.com*** and ***temporal.c1.mydomain.com*** you would set the value to **.c1.mydomain.com**.

For simplicity, the remaining defaults are fine. Commands to save and exit **nano**.
**Mac users:** ``` - to save your changes: Control + S - to exit: Control + X ``` **Windows users:** ``` - to save your changes: Ctrl + O - to exit: Ctrl + X ``` 5. Next, open the **docker-compose** file. ``` sudo nano docker-compose.yaml ``` The only changes that you should make here are to the AIS Platform image repositories. After opening the docker-compose file, scroll down to the Multiwoven Services and append **-ee** to the end of each repostiory and change the tag for each to **edge**. Before changes After changes 6. Deploy the AIS Platform. This step requires a private repository access key that you should have received from your AIS point of contact. If you do not have one, please reach out to AIS. ``` DOCKERHUB_USERNAME="multiwoven" DOCKERHUB_PASSWORD="YOUR_PRIVATE_ACCESS_TOKEN" sudo docker login --username $DOCKERHUB_USERNAME --password $DOCKERHUB_PASSWORD sudo docker-compose up -d ``` You can use the following command to ensure that none of the containers have exited ``` sudo docker ps -a ``` 7. Return to your browser and navigate back to the EC2 dashboard. In the left panel, scroll back down to **Target groups**. Click through each target group and verify that each has the registered instance showing as **healthy**. This may take a minute or two after starting the containers. 8. Once all target groups are showing your instance as healthy, you can navigate to your browser and enter the subdomain that you selected for the AIS Platform web app to get started! # AWS ECS Coming soon... # AWS EKS (Kubernetes) Coming soon... # Environment Variables Multiwoven uses the following environment variables for both the client and server: If you have any questions about these variables, please contact us at{" "} Hello Multiwoven or join our{" "} Slack Community. ## Required Variables `RAILS_ENV` - Rails Environment (development, test, production) `UI_HOST` - Hostname for UI service. Default is **localhost:8000** `API_HOST` - Hostname for API service. Default to **localhost:3000** `DB_HOST` - Database Host `DB_USERNAME` - Database Username `DB_PASSWORD` - Database Password `ALLOWED_HOST` - Frontend host that can connect to API. Prevents against DNS rebinding and other Host header attacks. Default values is localhost. `JWT_SECRET` - secret key used to sign generated token `USER_EMAIL_VERIFICATION` - Skip user email verification after signup.When set to true, ensure SMTP credentials are configured correctly so that verification emails can be sent to users. ## SMTP Configuration `SMTP_HOST` - This variable represents the host name of the SMTP server that the application will connect to for sending emails. The default configuration for SMTP\_HOST is set to `multiwoven.com`, indicating the server host. `SMTP_ADDRESS` - This environment variable specifies the server address where the SMTP service is hosted, critical for establishing a connection with the email server. Depending on the service provider, this address will vary. Here are examples of SMTP server addresses for some popular email providers: * Gmail: smtp.gmail.com - This is the server address for Google's Gmail service, allowing applications to send emails through Gmail's SMTP server. * Outlook: smtp-mail.outlook.com - This address is used for Microsoft's Outlook email service, enabling applications to send emails through Outlook's SMTP server. * Yahoo Mail: smtp.mail.yahoo.com - This address is used for Yahoo's SMTP server when configuring applications to send emails via Yahoo Mail. * AWS SES: *.*.amazonaws.com - This address format is used for AWS SES (Simple Email Service) SMTP servers when configuring applications to send emails via AWS SES. The specific region address should be used as shown in [here](https://docs.aws.amazon.com/general/latest/gr/ses.html) * Custom SMTP Server: mail.yourdomain.com - For custom SMTP servers, typically hosted by organizations or specialized email service providers, the SMTP address is specific to the domain or provider hosting the service. `SMTP_PORT` - This indicates the port number on which the SMTP server listens. The default configuration for SMTP\_PORT is set to 587, which is commonly used for SMTP with TLS/SSL. `SMTP_USERNAME` - This environment variable specifies the username required to authenticate with the SMTP server. This username could be an email address or a specific account identifier, depending on the requirements of the SMTP service provider being used (such as Gmail, Outlook, etc.). The username is essential for logging into the SMTP server to send emails. It is kept as an environment variable to maintain security and flexibility, allowing changes without code modification. `SMTP_PASSWORD` - Similar to the username, this environment variable holds the password associated with the SMTP\_USERNAME for authentication purposes. The password is critical for verifying the user's identity to the SMTP server, enabling the secure sending of emails. It is defined as an environment variable to ensure that sensitive credentials are not hard-coded into the application's source code, thereby protecting against unauthorized access and making it easy to update credentials securely. `SMTP_SENDER_EMAIL` - This variable specifies the email address that appears as the sender in the emails sent by the application. `BRAND_NAME` - This variable is used to customize the 'From' name in the emails sent from the application, allowing a personalized touch. It is set to **BRAND NAME**, which appears alongside the sender email address in outgoing emails. ## Sync Configuration `SYNC_EXTRACTOR_BATCH_SIZE` - Sync Extractor Batch Size `SYNC_LOADER_BATCH_SIZE` - Sync Loader Batch Size `SYNC_EXTRACTOR_THREAD_POOL_SIZE` - Sync Extractor Thread Pool Size `SYNC_LOADER_THREAD_POOL_SIZE` - Sync Loader Thread Pool Size ## Temporal Configuration `TEMPORAL_VERSION` - Temporal Version `TEMPORAL_UI_VERSION` - Temporal UI Version `TEMPORAL_HOST` - Temporal Host `TEMPORAL_PORT` - Temporal Port `TEMPORAL_ROOT_CERT` - Temporal Root Certificate `TEMPORAL_CLIENT_KEY` - Temporal Client Key `TEMPORAL_CLIENT_CHAIN` - Temporal Client Chain `TEMPORAL_POSTGRESQL_VERSION` - Temporal Postgres Version `TEMPORAL_POSTGRES_PASSWORD` - Temporal Postgres Password `TEMPORAL_POSTGRES_USER` - Temporal Postgres User `TEMPORAL_POSTGRES_DEFAULT_PORT` - Temporal Postgres Default Port `TEMPORAL_NAMESPACE` - Temporal Namespace `TEMPORAL_TASK_QUEUE` - Temporal Task Queue `TEMPORAL_ACTIVITY_THREAD_POOL_SIZE` - Temporal Activity Thread Pool Size `TEMPORAL_WORKFLOW_THREAD_POOL_SIZE` - Temporal Workflow Thread Pool Size ## Community Edition Configuration `VITE_API_HOST` - Hostname of API server `VITE_APPSIGNAL_PUSH_API_KEY` - AppSignal API key `VITE_BRAND_NAME` - Community Brand Name `VITE_LOGO_URL` - URL of Brand Logo `VITE_BRAND_COLOR` - Community Theme Color `VITE_BRAND_HOVER_COLOR` - Community Theme Color On Hover `VITE_FAV_ICON_URL` - URL of Brand Favicon ## Deployment Variables `APP_ENV` - Deployment environment. Default: community. `APP_REVISION` - Latest github commit sha. Used to identify revision of deployments. ## AWS Variables `AWS_ACCESS_KEY_ID` - AWS Access Key Id. Used to assume role in S3 connector. `AWS_SECRET_ACCESS_KEY` - AWS Secret Access Key. Used to assume role in S3 connector. ## Optional Variables `APPSIGNAL_PUSH_API_KEY` - API Key for AppSignal integration. `TRACK` - Track usage events. `NEW_RELIC_KEY` - New Relic Key `RAILS_LOG_LEVEL` - Rails log level. Default: info. # Google Cloud Compute Engine ## Deploying Multiwoven on Google Cloud Platform using Docker Compose This guide walks you through setting up Multiwoven, on a Google Cloud Platform (GCP) Compute Engine instance using Docker Compose. We'll cover launching the instance, installing Docker, running Multiwoven with its dependencies, and accessing the Multiwoven UI. **Prerequisites:** * A Google Cloud Platform account with an active project and billing enabled. * Basic knowledge of GCP, Docker, and command-line tools. * Docker Compose installed on your local machine. **Note:** This guide uses environment variables for sensitive information. Replace the placeholders with your own values before proceeding. **1. Create a GCP Compute Engine Instance:** 1. **Open the GCP Console:** [https://console.cloud.google.com](https://console.cloud.google.com) 2. **Navigate to Compute Engine:** Go to the "Compute Engine" section and click on "VM Instances." 3. **Create a new instance:** Choose an appropriate machine type based on your workload requirements. Ubuntu is a popular choice. 4. **Configure your instance:** * Select a suitable boot disk size and operating system image (Ubuntu recommended). * Enable SSH access with a strong password or SSH key. * Configure firewall rules to allow inbound traffic on port 22 (SSH) and potentially port 8000 (Multiwoven UI, optional). 5. **Create the instance:** Review your configuration and click "Create" to launch the instance. **2. Connect to your Instance:** 1. **Get the external IP address:** Once the instance is running, find its external IP address in the GCP Console. 2. **Connect via SSH:** Use your preferred SSH client to connect to the instance: ``` ssh -i your_key_pair.pem user@ ``` **3. Install Docker and Docker Compose:** 1. **Update and upgrade:** Run `sudo apt update && sudo apt upgrade -y` to ensure your system is up-to-date. 2. **Install Docker:** Follow the official Docker installation instructions for Ubuntu: [https://docs.docker.com/engine/install/](https://docs.docker.com/engine/install/) 3. **Install Docker Compose:** Download the latest version from the Docker Compose releases page and place it in a suitable directory (e.g., `/usr/local/bin/docker-compose`). Make the file executable: `sudo chmod +x /usr/local/bin/docker-compose`. 4. **Start and enable Docker:** Run `sudo systemctl start docker` and `sudo systemctl enable docker` to start Docker and configure it to start automatically on boot. **4. Download Multiwoven `docker-compose.yml` file and Configure Environment:** 1. **Download the file:**  ``` curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/docker-compose.yaml ``` 2. **Download the `.env` file:**   ``` curl -LO https://multiwoven-deployments.s3.amazonaws.com/docker/docker-compose/.env ``` 3. **Create and Configure `.env` File:** Rename `multiwoven/.env.example` to `.env`. This file holds environment variables for various services. Replace the placeholders with your own values, including:   \* `DB_PASSWORD` and `DB_USERNAME` for your PostgreSQL database   \* `REDIS_PASSWORD` for your Redis server   \* (Optional) Additional environment variables specific to your Multiwoven configuration **Example `.env` file:** ``` DB_PASSWORD=your_db_password DB_USERNAME=your_db_username REDIS_PASSWORD=your_redis_password # Modify your Multiwoven-specific environment variables here ``` **5. Run Multiwoven with Docker Compose:** **Start Multiwoven:** Navigate to the `multiwoven` directory and run. ```bash docker-compose up -d ``` **6. Accessing Multiwoven UI:** Open your web browser and navigate to `http://:8000` (replace `` with your instance's IP address). You should now see the Multiwoven UI. **7. Stopping Multiwoven:** To stop Multiwoven, navigate to the `multiwoven` directory and run. ```bash docker-compose down ``` **8. Upgrading Multiwoven:** When a new version of Multiwoven is released, you can upgrade the Multiwoven using the following command. ```bash docker-compose pull && docker-compose up -d ``` Make sure to run the above command from the same directory where the `docker-compose.yml` file is present. **Additional Notes:** **Note**: the frontend and backend services run on port 8000 and 3001, respectively. Make sure you update the **VITE\_API\_HOST** environment variable in the **.env** file to the desired backend service URL running on port 3000. * Depending on your firewall configuration, you might need to open port 8000 for inbound traffic. * For production deployments, consider using a managed load balancer and a Cloud SQL database instance for better performance and scalability. # Google Cloud GKE (Kubernetes) Coming soon... # Helm Charts ## Description: This helm chart is designed to deploy AI Squared's Platform 2.0 into a Kubernetes cluster. Platform 2.0 is cloud-agnostic and can be deployed successfully into any Kubernetes cluster, including clusters deployed via Azure Kubernetes Service, Elastic Kubernetes Service, Microk8s, etc. Along with the platform containers, there are also a couple of additional support resources added to simplify and further automate the installation process. These include: the **nginx-ingress resources** to expose the platform to end-users and **cert-manager** to automate the creation and renewal of TLS certificates. ### Coming Soon! We have a couple of useful features that are still in development that will further promote high availability, scalability and visibility into the platform pods! These features include **horizontal-pod autoscaling** based on pod CPU and memory utilization as well as in-cluster instances of both **Prometheus** and **Grafana**. ## Prerequisites: * Access to a DNS record set * Kubernetes cluster * [Install Kubernetes 1.16+](https://kubernetes.io/docs/tasks/tools/) * [Install Helm 3.1.0+](https://kubernetes.io/docs/tasks/tools/) * Temporal Namespace (optional) ## Overview of the Deployment Process 1. Install kubectl and helm on your local machine 2. Select required subdomains 3. Deploy the Cert-Manager Helm chart 4. Deploy the Multiwoven Helm Chart 5. Deploy additional (required) Nginx Ingress resources 6. Obtain the public IP address associated with your Nginx Ingress Controller 7. Create A records in your DNS record set that resolve to the public IP address of your Nginx Ingress Controller. 8. Wait for cert-manager to issue an invalid staging certificate to your K8s cluster 9. Switch letsencrypt-staging to letsencrypt-prod and upgrade Multiwoven again, this time obtaining a valid TLS certificate ## Installing Multiwoven via Helm Below is a shell script that can be used to deploy Multiwoven and its dependencies. ### Chart Dependencies #### Cert-Manager Cert-Manager is used to automatically request, implement and rotate TLS certificates for your deployment. Enabling TLS is required. #### Nginx-Ingress Nginx-Ingress resources are added to provide the Multiwoven Ingress Controller with a external IP address. ### Install Multiwoven #### Environment Variables: ##### Generic 1. tls-admin-email-address -> the email address that will receive email notifications about pending automatic TLS certificate rotations 2. api-host -> api.your\_domain (ex. api.multiwoven.com) 3. ui-host -> app.your\_domain (ex. app.multiwoven.com) ##### Temporal - Please read the [Notes](#notes) section below 4. temporal-ui-host -> temporal.your\_domain (ex. temporal.multiwoven.com) 5. temporalHost -> your Temporal Cloud host name (ex. my.personal.tmprl.cloud) 6. temporalNamespace -> your Temporal Namespace, verify within your Temporal Cloud account (ex. my.personal) #### Notes: * Deploying with the default In-cluster Temporal (recommended for Development workloads): 1. Only temporal-ui-host is required. You should leave multiwovenConfig.temporalHost, temporal.enabled and multiwovenConfig.temporalNamespace commented out. You should also leave the temporal-cloud secret commented out as well. * Deploying with Temporal Cloud (HIGHLY recommended for Production workloads): 1. comment out or remove the flag setting multiwovenConfig.temporalUiHost 2. Uncomment the flags setting multiwovenConfig.temporalHost, temporal.enabled and multiwovenConfig.temporalNamespace. Also uncomment the temporal-cloud secret. 3. Before running this script, you need to make sure that your Temporal Namespace authentication certificate key and pem files are in the same directory as the script. We recommend renaming these files to temporal.key and temporal.pem for simplicity. * Notice that for tlsCertIssuer, the value letsencrypt-staging is present. When the intial installation is done and cert-manager has successfully issued an invalid certificate for your 3 subdomains, you will switch this value to letsencrypt-prod to obtain a valid certificate. It is very important that you follow the steps written out here as LetsEncrypt's production server only allows 5 attempts per week to obtain a valid certificate. This switch should be done LAST after you have verified that everything is already working as expected. ``` #### Pull and deploy the cert-manager Helm chart cd charts/multiwoven echo "installing cert-manager" helm repo add jetstack https://charts.jetstack.io --force-update helm repo update helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.14.5 --set installCRDs=true #### Pull and deploy the Multiwoven Helm chart echo "installing Multiwoven" helm repo add multiwoven https://multiwoven.github.io/helm-charts helm upgrade -i multiwoven multiwoven/multiwoven \ --set multiwovenConfig.tlsAdminEmail= \ --set multiwovenConfig.tlsCertIssuer=letsencrypt-staging \ --set multiwovenConfig.apiHost= \ --set multiwovenConfig.uiHost= \ --set multiwovenWorker.multiwovenWorker.args={./app/temporal/cli/worker} \ --set multiwovenConfig.temporalUiHost= # --set temporal.enabled=false \ # --set multiwovenConfig.temporalHost= \ # --set multiwovenConfig.temporalNamespace= # kubectl create secret generic temporal-cloud -n multiwoven \ # --from-file=temporal-root-cert=./temporal.pem \ # --from-file=temporal-client-key=./temporal.key # Install additional required Nginx ingress resources echo "installing ingress-nginx resources" kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml ``` #### Post Installation Steps 1. Run the following command to find the external IP address of your Ingress Controller. Note that it may take a minute or two for this value to become available post installation. ``` kubectl get svc -n ingress-nginx ``` 2. Once you have this IP address, go to your DNS record set and use that IP address to create three A records, one for each subdomain. Below are a list of Cloud Service Provider DNS tools but please refer to the documentation of your specific provider if not listed below. * [Adding a new record in Azure DNS Zones](https://learn.microsoft.com/en-us/azure/dns/dns-operations-recordsets-portal) * [Adding a new record in AWS Route 53](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-creating.html) * [Adding a new record in GCP Cloud DNS](https://cloud.google.com/dns/docs/records) 3. Run the following command, repeatedly, until an invalid LetsEncrypt staging certificate has been issued for your Ingress Controller. ``` kubectl describe certificate -n multiwoven mw-tls-cert ``` When the certificate has been issued, you will see the following output from the command above. We also encourage you to further verify by navigating to your subdomain, app.your\_domain, and check the certificate received by the browser. You should see something similar to the image below: Once the invalid certificate has been successfully issued, you are ready for the final steps. 4. Edit the shell script above by changing the tlsCertIssuer value from letsencrypt-staging to letsencrypt-prod and run the script again. Do not worry when you see Installation Failed for cert-manager, you are seeing this because it was installed on the intial run. 5. Repeat Post Installation Step 3 until a valid certificate has been issued. Once issued, your deployment is complete and you can navigate to app.your\_domain to get started using Mutliwoven! Happy Helming! ## Helm Chart Environment Values ### Multiwoven Helm Configuration #### General Configuration * **kubernetesClusterDomain**: The domain used within the Kubernetes cluster. * Default: `cluster.local` * **kubernetesNamespace**: The Kubernetes namespace for deployment. * Default: `multiwoven` #### Multiwoven Configuration | Parameter | Description | Default | | ------------------------------------------------- | ----------------------------------------------------------- | --------------------------------------------- | | `multiwovenConfig.apiHost` | Hostname for the API service. | `api.multiwoven.com` | | `multiwovenConfig.appEnv` | Deployment environment. | `community` | | `multiwovenConfig.appRevision` | Latest github commit sha, identifies revision of deployment | \`\` | | `multiwovenConfig.appsignalPushApiKey` | AppSignal API key. | `yourkey` | | `multiwovenConfig.awsAccessKeyId` | AWS Access Key Id. Used to assume role in S3 connector. | \`\` | | `multiwovenConfig.awsSecretAccessKey` | AWS Secret Access Key. Used to assume role in S3 connector. | \`\` | | `multiwovenConfig.dbHost` | Hostname for the PostgreSQL database service. | `multiwoven-postgresql` | | `multiwovenConfig.dbPassword` | Password for the database user. | `password` | | `multiwovenConfig.dbPort` | Port on which the database service is running. | `5432` | | `multiwovenConfig.dbUsername` | Username for the database. | `multiwoven` | | `multiwovenConfig.grpcEnableForkSupport` | GRPC\_ENABLE\_FORK\_SUPPORT env variable. | `1` | | `multiwovenConfig.newRelicKey` | New Relic License Key. | `yourkey` | | `multiwovenConfig.railsEnv` | Rails environment setting. | `development` | | `multiwovenConfig.railsLogLevel` | Rails log level. | `info` | | `multiwovenConfig.smtpAddress` | SMTP server address. | `smtp.yourdomain.com` | | `multiwovenConfig.smtpBrandName` | SMTP brand name used in From email. | `Multiwoven` | | `multiwovenConfig.smtpHost` | SMTP server host. | `yourdomain.com` | | `multiwovenConfig.smtpPassword` | SMTP server password. | `yourpassword` | | `multiwovenConfig.smtpPort` | SMTP server port. | `587` | | `multiwovenConfig.smtpUsername` | SMTP server username. | `yourusername` | | `multiwovenConfig.smtpSenderEmail` | SMTP sender email. | `admin@yourdomain.com` | | `multiwovenConfig.snowflakeDriverPath` | Path to the Snowflake ODBC driver. | `/usr/lib/snowflake/odbc/lib/libSnowflake.so` | | `multiwovenConfig.syncExtractorBatchSize` | Batch size for the sync extractor. | `1000` | | `multiwovenConfig.syncExtractorThreadPoolSize` | Thread pool size for the sync extractor. | `10` | | `multiwovenConfig.syncLoaderBatchSize` | Batch size for the sync loader. | `1000` | | `multiwovenConfig.syncLoaderThreadPoolSize` | Thread pool size for the sync loader. | `10` | | `multiwovenConfig.temporalActivityThreadPoolSize` | Thread pool size for Temporal activities. | `20` | | `multiwovenConfig.temporalClientChain` | Path to Temporal client chain certificate. | `/certs/temporal.chain.pem` | | `multiwovenConfig.temporalClientKey` | Path to Temporal client key. | `/certs/temporal.key` | | `multiwovenConfig.temporalHost` | Hostname for Temporal service. | `multiwoven-temporal` | | `multiwovenConfig.temporalNamespace` | Namespace for Temporal service. | `multiwoven-dev` | | `multiwovenConfig.temporalPort` | Port for Temporal service. | `7233` | | `multiwovenConfig.temporalPostgresDefaultPort` | Default port for Temporal's PostgreSQL database. | `5432` | | `multiwovenConfig.temporalPostgresPassword` | Password for Temporal's PostgreSQL database. | `password` | | `multiwovenConfig.temporalPostgresUser` | Username for Temporal's PostgreSQL database. | `multiwoven` | | `multiwovenConfig.temporalPostgresqlVersion` | PostgreSQL version for Temporal. | `13` | | `multiwovenConfig.temporalRootCert` | Path to Temporal root certificate. | `/certs/temporal.pem` | | `multiwovenConfig.temporalTaskQueue` | Task queue for Temporal workflows. | `sync-dev` | | `multiwovenConfig.temporalUiVersion` | Version of Temporal UI. | `2.23.2` | | `multiwovenConfig.temporalVersion` | Version of Temporal service. | `1.22.4` | | `multiwovenConfig.temporalWorkflowThreadPoolSize` | Thread pool size for Temporal workflows. | `10` | | `multiwovenConfig.uiHost` | UI host for the application interface. | `app.multiwoven.com` | | `multiwovenConfig.viteApiHost` | API host for the web application. | `api.multiwoven.com` | | `multiwovenConfig.viteAppsignalPushApiKey` | AppSignal API key. | `yourkey` | | `multiwovenConfig.viteBrandName` | Community Brand Name. | `Multiwoven` | | `multiwovenConfig.viteLogoUrl` | URL of Brand Logo. | | | `multiwovenConfig.viteBrandColor` | Community Theme Color. | | | `multiwovenConfig.viteBrandHoverColor` | Community Theme Color On Hover. | | | `multiwovenConfig.viteFavIconUrl` | URL of Brand Favicon. | | | 'multiwovenConfig.workerHost\` | Worker host for the worker service. | 'worker.multiwoven.com' | ### Multiwoven PostgreSQL Configuration | Parameter | Description | Default | | ------------------------------------------------ | -------------------------------------------------- | ----------- | | `multiwovenPostgresql.enabled` | Whether or not to deploy PostgreSQL. | `true` | | `multiwovenPostgresql.image.repository` | Docker image repository for PostgreSQL. | `postgres` | | `multiwovenPostgresql.image.tag` | Docker image tag for PostgreSQL. | `13` | | `multiwovenPostgresql.resources.limits.cpu` | CPU resource limits for PostgreSQL pod. | `1` | | `multiwovenPostgresql.resources.limits.memory` | Memory resource limits for PostgreSQL pod. | `2Gi` | | `multiwovenPostgresql.resources.requests.cpu` | CPU resource requests for PostgreSQL pod. | `500m` | | `multiwovenPostgresql.resources.requests.memory` | Memory resource requests for PostgreSQL pod. | `1Gi` | | `multiwovenPostgresql.ports.name` | Port name for PostgreSQL service. | `postgres` | | `multiwovenPostgresql.ports.port` | Port number for PostgreSQL service. | `5432` | | `multiwovenPostgresql.ports.targetPort` | Target port for PostgreSQL service within the pod. | `5432` | | `multiwovenPostgresql.replicas` | Number of PostgreSQL pod replicas. | `1` | | `multiwovenPostgresql.type` | Service type for PostgreSQL. | `ClusterIP` | ### Multiwoven Server Configuration | Parameter | Description | Default | | -------------------------------------------- | --------------------------------------------------------- | ------------------------------ | | `multiwovenServer.image.repository` | Docker image repository for Multiwoven server. | `multiwoven/multiwoven-server` | | `multiwovenServer.image.tag` | Docker image tag for Multiwoven server. | `latest` | | `multiwovenServer.resources.limits.cpu` | CPU resource limits for Multiwoven server pod. | `2` | | `multiwovenServer.resources.limits.memory` | Memory resource limits for Multiwoven server pod. | `2Gi` | | `multiwovenServer.resources.requests.cpu` | CPU resource requests for Multiwoven server pod. | `1` | | `multiwovenServer.resources.requests.memory` | Memory resource requests for Multiwoven server pod. | `1Gi` | | `multiwovenServer.ports.name` | Port name for Multiwoven server service. | `3000` | | `multiwovenServer.ports.port` | Port number for Multiwoven server service. | `3000` | | `multiwovenServer.ports.targetPort` | Target port for Multiwoven server service within the pod. | `3000` | | `multiwovenServer.replicas` | Number of Multiwoven server pod replicas. | `1` | | `multiwovenServer.type` | Service type for Multiwoven server. | `ClusterIP` | ### Multiwoven Worker Configuration | Parameter | Description | Default | | -------------------------------------------- | --------------------------------------------------------- | ------------------------------ | | `multiwovenWorker.args` | Command arguments for the Multiwoven worker. | See value | | `multiwovenWorker.healthPort` | The port in which the health check endpoint is exposed. | `4567` | | `multiwovenWorker.image.repository` | Docker image repository for Multiwoven worker. | `multiwoven/multiwoven-server` | | `multiwovenWorker.image.tag` | Docker image tag for Multiwoven worker. | `latest` | | `multiwovenWorker.resources.limits.cpu` | CPU resource limits for Multiwoven worker pod. | `1` | | `multiwovenWorker.resources.limits.memory` | Memory resource limits for Multiwoven worker pod. | `1Gi` | | `multiwovenWorker.resources.requests.cpu` | CPU resource requests for Multiwoven worker pod. | `500m` | | `multiwovenWorker.resources.requests.memory` | Memory resource requests for Multiwoven worker pod. | `512Mi` | | `multiwovenWorker.ports.name` | Port name for Multiwoven worker service. | `4567` | | `multiwovenWorker.ports.port` | Port number for Multiwoven worker service. | `4567` | | `multiwovenWorker.ports.targetPort` | Target port for Multiwoven worker service within the pod. | `4567` | | `multiwovenWorker.replicas` | Number of Multiwoven worker pod replicas. | `1` | | `multiwovenWorker.type` | Service type for Multiwoven worker. | `ClusterIP` | ### Persistent Volume Claim (PVC) | Parameter | Description | Default | | -------------------- | --------------------------------- | ------- | | `pvc.storageRequest` | Storage request size for the PVC. | `100Mi` | ### Temporal Configuration | Parameter | Description | Default | | --------------------------------------------- | ---------------------------------------------------------- | ----------------------- | | `temporal.enabled` | Whether or not to deploy Temporal and Temporal UI service. | `true` | | `temporal.ports.name` | Port name for Temporal service. | `7233` | | `temporal.ports.port` | Port number for Temporal service. | `7233` | | `temporal.ports.targetPort` | Target port for Temporal service within the pod. | `7233` | | `temporal.replicas` | Number of Temporal service pod replicas. | `1` | | `temporal.temporal.env.db` | Database type for Temporal. | `postgresql` | | `temporal.temporal.image.repository` | Docker image repository for Temporal. | `temporalio/auto-setup` | | `temporal.temporal.image.tag` | Docker image tag for Temporal. | `1.22.4` | | `temporal.temporal.resources.limits.cpu` | CPU resource limits for Temporal pod. | `1` | | `temporal.temporal.resources.limits.memory` | Memory resource limits for Temporal pod. | `2Gi` | | `temporal.temporal.resources.requests.cpu` | CPU resource requests for Temporal pod. | `500m` | | `temporal.temporal.resources.requests.memory` | Memory resource requests for Temporal pod. | `1Gi` | | `temporal.type` | Service type for Temporal. | `ClusterIP` | ### Temporal UI Configuration | Parameter | Description | Default | | ---------------------------------------------------- | --------------------------------------------------------------- | -------------------------- | | `temporalUi.ports.name` | Port name for Temporal UI service. | `8080` | | `temporalUi.ports.port` | Port number for Temporal UI service. | `8080` | | `temporalUi.ports.targetPort` | Target port for Temporal UI service within the pod. | `8080` | | `temporalUi.replicas` | Number of Temporal UI service pod replicas. | `1` | | `temporalUi.temporalUi.env.temporalAddress` | Temporal service address for UI. | `multiwoven-temporal:7233` | | `temporalUi.temporalUi.env.temporalAuthCallbackUrl` | Authentication/authorization callback URL. | | | `temporalUi.temporalUi.env.temporalAuthClientId` | Authentication/authorization client ID. | | | `temporalUi.temporalUi.env.temporalAuthClientSecret` | Authentication/authorization client secret. | | | `temporalUi.temporalUi.env.temporalAuthEnabled` | Enable or disable authentication/authorization for Temporal UI. | `false` | | `temporalUi.temporalUi.env.temporalAuthProviderUrl` | Authentication/authorization OIDC provider URL. | | | `temporalUi.temporalUi.env.temporalCorsOrigins` | Allowed CORS origins for Temporal UI. | `http://localhost:3000` | | `temporalUi.temporalUi.env.temporalUiPort` | Port for Temporal UI service. | `8080` | | `temporalUi.temporalUi.image.repository` | Docker image repository for Temporal UI. | `temporalio/ui` | | `temporalUi.temporalUi.image.tag` | Docker image tag for Temporal UI. | `2.22.3` | | `temporalUi.type` | Service type for Temporal UI. | `ClusterIP` | # Heroku Coming soon... # OpenShift Coming soon... # Multiwoven > Multiwoven is AI Squared's **Open Source Reverse ETL** platform that turns any data warehouse into a Customer Data Platform (CDP). Hero Light Hero Dark ## Getting started Uncover everything about Reverse ETL and its role in activating data from warehouses to various destinations. Turns any data warehouse (like Snowflake, Redshift, BigQuery, DataBricks, Postgres) into a Customer Data Platform (CDP) The first step is to set up the source of the data. This could be a data warehouse, a database, or any other source of data. Head over to the [source setup](/sources/overview) guide to get started. The second step is to set up the destination for the selected source. This could be a CRM, a marketing automation platform, or any other destination. Head over to the [destination setup](/destinations/overview) guide to get started. The final step is to model the data and sync it from the source to the destination. This is where you define the data model and the sync schedule. Head over to the [data modeling](/models/overview) and [sync](/sync/overview) guides to get started. ## Popular destinations } href="/destinations/crm/salesforce"> Salesforce is a popular destination for customer relationship management. } href="/destinations/adtech/facebook-ads" > Facebook Ads is a popular destination for marketing and advertising. } href="/destinations/team-collaboration/slack"> Slack is a popular destination for team collaboration and communication. } href="/destinations/productivity-tools/google-sheets"> Google Sheets is a popular destination for team collaboration. # 2024 releases Version: v0.25.0 to v0.30.0 Version: v0.23.0 to v0.24.0 Version: v0.20.0 to v0.22.0 Version: v0.14.0 to v0.19.0 Version: v0.12.0 to v0.13.0 Version: v0.5.0 to v0.8.0 # August 2024 releases Release updates for the month of August ## 🚀 **New Features** ### 🔄 **Enable/Disable Sync** We’ve introduced the ability to enable or disable a sync. When a sync is disabled, it won’t execute according to its schedule, allowing you to effectively pause it without the need to delete it. This feature provides greater control and flexibility in managing your sync operations. ### 🧠 **Source: Databricks AI Model Connector** Multiwoven now integrates seamlessly with [Databricks AI models](https://docs.squared.ai/guides/data-integration/sources/databricks-model) in the source connectors. This connection allows users to activate AI models directly through Multiwoven, enhancing your data processing and analytical capabilities with cutting-edge AI tools. ### 📊 **Destination: Microsoft Excel** You can now use [Microsoft Excel](https://docs.squared.ai/guides/data-integration/destinations/productivity-tools/microsoft-excel) as a destination connector. Deliver your modeled data directly to Excel sheets for in-depth analysis or reporting. This addition simplifies workflows for those who rely on Excel for their data presentation and analysis needs. ### ✅ **Triggering Test Sync** Before running a full sync, users can now initiate a test sync to verify that everything is functioning as expected. This feature ensures that potential issues are caught early, saving time and resources. ### 🏷️ **Sync Run Type** Sync types are now clearly labeled as either "General" or "Test" in the Syncs Tab. This enhancement provides clearer context for each sync operation, making it easier to distinguish between different sync runs. ### 🛢️ **Oracle DB as a Destination Connector** [Oracle DB](https://docs.squared.ai/guides/data-integration/destinations/database/oracle) is now available as a destination connector. Users can navigate to **Add Destination**, select **Oracle**, and input the necessary database details to route data directly to Oracle databases. ### 🗄️ **Oracle DB as a Source Connector** [Oracle DB](https://docs.squared.ai/guides/data-integration/sources/oracle) has also been added as a source connector. Users can pull data from Oracle databases by navigating to **Add Source**, selecting **Oracle**, and entering the database details. *** ## 🔧 **Improvements** ### **Memory Bloat Issue in Sync** Resolved an issue where memory bloat was affecting sync performance over time, ensuring more stable and efficient sync operations. ### **Discover and Table URL Fix** Fixed issues with discovering and accessing table URLs, enhancing the reliability and accuracy of data retrieval processes. ### **Disable to Fields** Added the option to disable fields where necessary, giving users more customization options to fit their specific needs. ### **Query Source Response Update** Updated the query source response mechanism, improving data handling and accuracy in data query operations. ### **OCI8 Version Fix** Resolved issues related to the OCI8 version, ensuring better compatibility and smoother database interactions. ### **User Read Permission Update** Updated user read permissions to enhance security and provide more granular control over data access. ### **Connector Name Update** Updated connector names across the platform to ensure better clarity and consistency, making it easier to manage and understand your integrations. ### **Account Verification Route Removal** Streamlined the user signup process by removing the account verification route, reducing friction for new users. ### **Connector Creation Process** Refined the connector creation process, making it more intuitive and user-friendly, thus reducing the learning curve for new users. ### **README Update** The README file has been updated to reflect the latest changes and enhancements, providing more accurate and helpful guidance. ### **Request/Response Logs Added** We’ve added request/response logs for multiple connectors, including Klaviyo, HTTP, Airtable, Slack, MariaDB, Google Sheets, Iterable, Zendesk, HubSpot, Stripe, and Salesforce CRM, improving debugging and traceability. ### **Logger Issue in Sync** Addressed a logging issue within sync operations, ensuring that logs are accurate and provide valuable insights. ### **Main Layout Protected** Wrapped the main layout with a protector, enhancing security and stability across the platform. ### **User Email Verification** Implemented email verification during signup using Devise, increasing account security and ensuring that only verified users have access. ### **Databricks Datawarehouse Connector Name Update** Renamed the Databricks connection to "Databricks Datawarehouse" for improved clarity and better alignment with user expectations. ### **Version Upgrade to 0.9.1** The platform has been upgraded to version `0.9.1`, incorporating all the above features and improvements, ensuring a more robust and feature-rich experience. ### **Error Message Refactoring** Refactored error messages to align with agreed-upon standards, resulting in clearer and more consistent communication across the platform. # July 2024 releases Release updates for the month of July ## ✨ **New Features** ### 🔍 **Search Filter in Table Selector** The table selector method now includes a powerful search filter. This feature enhances your workflow by allowing you to swiftly locate and select the exact tables you need, even in large datasets. It’s all about saving time and boosting productivity. ### 🏠 **Databricks Lakehouse Destination** We're excited to introduce Databricks Lakehouse as a new destination connector. Seamlessly integrate your data pipelines with Databricks Lakehouse, harnessing its advanced analytics capabilities for data processing and AI-driven insights. This feature empowers your data strategies with greater flexibility and power. ### 📅 **Manual Sync Schedule Controller** Take control of your data syncs with the new Manual Sync Schedule controller. This feature gives you the freedom to define when and how often syncs occur, ensuring they align perfectly with your business needs while optimizing resource usage. ### 🛢️ **MariaDB Destination Connector** MariaDB is now available as a destination connector! You can now channel your processed data directly into MariaDB databases, enabling robust data storage and processing workflows. This integration is perfect for users operating in MariaDB environments. ### 🎛️ **Table Selector and Layout Enhancements** We’ve made significant improvements to the table selector and layout. The interface is now more intuitive, making it easier than ever to navigate and manage your tables, especially in complex data scenarios. ### 🔄 **Catalog Refresh** Introducing on-demand catalog refresh! Keep your data sources up-to-date with a simple refresh, ensuring you always have the latest data structure available. Say goodbye to outdated data and hello to consistency and accuracy. ### 🛡️ **S3 Connector ARN Support for Authentication** Enhance your security with ARN (Amazon Resource Name) support for Amazon S3 connectors. This update provides a more secure and scalable approach to managing access to your S3 resources, particularly beneficial for large-scale environments. ### 📊 **Integration Changes for Sync Record Log** We’ve optimized the integration logic for sync record logs. These changes ensure more reliable logging, making it easier to track sync operations and diagnose issues effectively. ### 🗄️ **Server Changes for Log Storage in Sync Record Table** Logs are now stored directly in the sync record table, centralizing your data and improving log accessibility. This update ensures that all relevant sync information is easily retrievable for analysis. ### ✅ **Select Row Support in Data Table** Interact with your data tables like never before! We've added row selection support, allowing for targeted actions such as editing or deleting entries directly from the table interface. ### 🛢️ **MariaDB Source Connector** The MariaDB source connector is here! Pull data directly from MariaDB databases into Multiwoven for seamless integration into your data workflows. ### 🛠️ **Sync Records Error Log** A detailed error log feature has been added to sync records, providing granular visibility into issues that occur during sync operations. Troubleshooting just got a whole lot easier! ### 🛠️ **Model Query Type - Table Selector** The table selector is now available as a model query type, offering enhanced flexibility in defining queries and working with your data models. ### 🔄 **Force Catalog Refresh** Set the refresh flag to true, and the catalog will be forcefully refreshed. This ensures you're always working with the latest data, reducing the chances of outdated information impacting your operations. ## 🔧 **Improvements** * **Manual Sync Delete API Call**: Enhanced the API call for deleting manual syncs for smoother operations. * **Server Error Handling**: Improved error handling to better display server errors when data fetches return empty results. * **Heartbeat Timeout in Extractor**: Introduced new actions to handle heartbeat timeouts in extractors for improved reliability. * **Sync Run Type Column**: Added a `sync_run_type` column in sync logs for better tracking and operational clarity. * **Refactor Discover Stream**: Refined the discover stream process, leading to better efficiency and reliability. * **DuckDB HTTPFS Extension**: Introduced server installation steps for the DuckDB `httpfs` extension. * **Temporal Initialization**: Temporal processes are now initialized in all registered namespaces, improving system stability. * **Password Reset Email**: Updated the reset password email template and validation for a smoother user experience. * **Organization Model Changes**: Applied structural changes to the organization model, enhancing functionality. * **Log Response Validation**: Added validation to log response bodies, improving error detection. * **Missing DuckDB Dependencies**: Resolved missing dependencies for DuckDB, ensuring smoother operations. * **STS Client Initialization**: Removed unnecessary credential parameters from STS client initialization, boosting security. * **Main Layout Error Handling**: Added error screens for the main layout to improve user experience when data is missing or errors occur. * **Server Gem Updates**: Upgraded server gems to the latest versions, enhancing performance and security. * **AppSignal Logging**: Enhanced AppSignal logging by including app request and response logs for better monitoring. * **Sync Records Table**: Added a dedicated table for sync records to improve data management and retrieval. * **AWS S3 Connector**: Improved handling of S3 credentials and added support for STS credentials in AWS S3 connectors. * **Sync Interval Dropdown Fix**: Fixed an issue where the sync interval dropdown text was hidden on smaller screens. * **Form Data Processing**: Added a pre-check process for form data before checking connections, improving validation and accuracy. * **S3 Connector ARN Support**: Updated the gem to support ARN-based authentication for S3 connectors, enhancing security. * **Role Descriptions**: Updated role descriptions for clearer understanding and easier management. * **JWT Secret Configuration**: JWT secret is now configurable from environment variables, boosting security practices. * **MariaDB README Update**: Updated the README file to include the latest information on MariaDB connectors. * **Logout Authorization**: Streamlined the logout process by skipping unnecessary authorization checks. * **Sync Record JSON Error**: Added a JSON error field in sync records to enhance error tracking and debugging. * **MariaDB DockerFile Update**: Added `mariadb-dev` to the DockerFile to better support MariaDB integrations. * **Signup Error Response**: Improved the clarity and detail of signup error responses. * **Role Policies Update**: Refined role policies for enhanced access control and security. * **Pundit Policy Enhancements**: Applied Pundit policies at the role permission level, ensuring robust authorization management. # June 2024 releases Release updates for the month of June # 🚀 New Features * **Iterable Destination Connector**\ Integrate with Iterable, allowing seamless data flow to this popular marketing automation platform. * **Workspace Settings and useQueryWrapper**\ New enhancements to workspace settings and the introduction of `useQueryWrapper` for improved data handling. * **Amazon S3 Source Connector**\ Added support for Amazon S3 as a source connector, enabling data ingestion directly from your S3 buckets. # 🛠️ Improvements * **GitHub URL Issues**\ Addressed inconsistencies with GitHub URLs in the application. * **Change GitHub PAT to SSH Private Key**\ Updated authentication method from GitHub PAT to SSH Private Key for enhanced security. * **UI Maintainability and Workspace ID on Page Refresh**\ Improved UI maintainability and ensured that the workspace ID persists after page refresh. * **CE Sync Commit for Multiple Commits**\ Fixed the issue where CE sync commits were not functioning correctly for multiple commits. * **Add Role in User Info API Response**\ Enhanced the user info API to include role details in the response. * **Sync Write Update Action for Destination**\ Synchronized the write update action across various destinations for consistency. * **Fix Sync Name Validation Error**\ Resolved validation errors in sync names due to contract issues. * **Update Commit Message Regex**\ Updated the regular expression for commit messages to follow git conventions. * **Update Insert and Update Actions**\ Renamed `insert` and `update` actions to `destination_insert` and `destination_update` for clarity. * **Comment Contract Valid Rule in Update Sync Action**\ Adjusted the contract validation rule in the update sync action to prevent failures. * **Fix for Primary Key in `destination_update`**\ Resolved the issue where `destination_update` was not correctly picking up the primary key. * **Add Limit and Offset Query Validator**\ Introduced validation for limit and offset queries to improve API reliability. * **Ignore RBAC for Get Workspaces API**\ Modified the API to bypass Role-Based Access Control (RBAC) for fetching workspaces. * **Heartbeat Timeout Update for Loader**\ Updated the heartbeat timeout for the loader to ensure smoother operations. * **Add Strong Migration Gem**\ Integrated the Strong Migration gem to help with safe database migrations. Stay tuned for more exciting updates in the upcoming releases! # May 2024 releases Release updates for the month of May # 🚀 New Features * **Role and Resource Migration**\ Introduced migration capabilities for roles and resources, enhancing data management and security. * **Zendesk Destination Connector**\ Added support for Zendesk as a destination connector, enabling seamless integration with Zendesk for data flow. * **Athena Connector**\ Integrated the Athena Connector, allowing users to connect to and query Athena directly from the platform. * **Support for Temporal Cloud**\ Enabled support for Temporal Cloud, facilitating advanced workflow orchestration in the cloud. * **Workspace APIs for CE**\ Added Workspace APIs for the Community Edition, expanding workspace management capabilities. * **HTTP Destination Connector**\ Introduced the HTTP Destination Connector, allowing data to be sent to any HTTP endpoint. * **Separate Routes for Main Application**\ Organized and separated routes for the main application, improving modularity and maintainability. * **Compression Support for SFTP**\ Added compression support for SFTP, enabling faster and more efficient data transfers. * **Password Field Toggle**\ Introduced a toggle to view or hide password field values, enhancing user experience and security. * **Dynamic UI Schema Generation**\ Added dynamic generation of UI schemas, streamlining the user interface customization process. * **Health Check Endpoint for Worker**\ Added a health check endpoint for worker services, ensuring better monitoring and reliability. * **Skip Rows in Sync Runs Table**\ Implemented functionality to skip rows in the sync runs table, providing more control over data synchronization. * **Cron Expression as Schedule Type**\ Added support for using cron expressions as a schedule type, offering more flexibility in task scheduling. * **SQL Autocomplete**\ Introduced SQL autocomplete functionality, improving query writing efficiency. # 🛠️ Improvements * **Text Update in Finalize Source Form**\ Changed and improved the text in the Finalize Source Form for clarity. * **Rate Limiter Spec Failure**\ Fixed a failure issue in the rate limiter specifications, ensuring better performance and stability. * **Check for Null Record Data**\ Added a condition to check if record data is null, preventing errors during data processing. * **Cursor Field Mandatory Check**\ Ensured that the cursor field is mandatory, improving data integrity during synchronization. * **Docker Build for ARM64 Release**\ Fixed the Docker build process for ARM64 releases, ensuring compatibility across architectures. * **UI Auto Deploy**\ Improved the UI auto-deployment process for more efficient updates. * **Cursor Query for SOQL**\ Added support for cursor queries in SOQL, enhancing Salesforce data operations. * **Skip Cursor Query for Empty Cursor Field**\ Implemented a check to skip cursor queries when the cursor field is empty, avoiding unnecessary processing. * **Updated Integration Gem Version**\ Updated the integration gem to version 0.1.67, including support for Athena source, Zendesk, and HTTP destinations. * **Removed Stale User Management APIs**\ Deleted outdated user management APIs and made changes to role ID handling for better security. * **Color and Logo Theme Update**\ Changed colors and logos to align with the new theme, providing a refreshed UI appearance. * **Refactored Modeling Method Screen**\ Refactored the modeling method screen for better usability and code maintainability. * **Removed Hardcoded UI Schema**\ Removed hardcoded UI schema elements, making the UI more dynamic and adaptable. * **Heartbeat Timeout for Loader**\ Updated the heartbeat timeout for the loader, improving the reliability of the loading process. * **Integration Gem to 1.63**\ Bumped the integration gem version to 1.63, including various improvements and bug fixes. * **Core Chakra Config Update**\ Updated the core Chakra UI configuration to support new branding requirements. * **Branding Support in Config**\ Modified the configuration to support custom branding, allowing for more personalized user experiences. * **Strong Migration Gem Addition**\ Integrated the Strong Migration gem to ensure safer and more efficient database migrations. Stay tuned for more exciting updates in future releases! # October 2024 releases Release updates for the month of October # 🚀 New Features * **Data Apps Configurations and Rendering**\ Provides robust configurations and rendering capabilities for data apps, enhancing customization. * **Scale and Text Input Feedback Methods**\ Introduces new feedback options with scale and text inputs to capture user insights effectively. * **Support for Multiple Visual Components**\ Expands visualization options by supporting multiple visual components, enriching data presentation. * **Audit Log Filter**\ Adds a filter feature in the Audit Log, simplifying the process of finding specific entries. *** # 🛠 Improvements * **Disable Mixpanel Tracking**\ Disabled Mixpanel tracking for enhanced data privacy and user control. * **Data App Runner Script URL Fix**\ Resolved an issue with the UI host URL in the data app runner script for smoother operation. * **Text Input Bugs**\ Fixed bugs affecting text input functionality, improving stability and responsiveness. * **Dynamic Variables in Naming and Filters**\ Adjusted naming conventions and filters to rely exclusively on dynamic variables, increasing flexibility and reducing redundancy. * **Sort Data Apps List in Descending Order**\ The data apps list is now sorted in descending order by default for easier access to recent entries. * **Data App Response Enhancements**\ Updated responses for data app creation and update APIs, improving clarity and usability. *** > For further details on any feature or update, check the detailed documentation or contact our support team. We’re here to help make your experience seamless! *** # September 2024 releases Release updates for the month of September # 🚀 New Features * **AI/ML Sources**\ Introduces support for a range of AI/ML sources, broadening model integration capabilities. * **Added AI/ML Models Support**\ Comprehensive support for integrating and managing AI and ML models across various workflows. * **Data App Update API**\ This API endpoint allows users to update existing data apps without needing to recreate them from scratch. By enabling seamless updates with the latest configurations and features, users can Save time, Improve accuracy and Ensure consistency * \*\*Donut Chart Component The donut chart component enhances data visualization by providing a clear, concise way to represent proportions or percentages within a dataset. * **Google Vertex Model Source Connector**\ Enables connection to Google Vertex AI, expanding options for model sourcing and integration. *** # 🛠️ Improvements * **Verify User After Signup**\ A new verification step ensures all users are authenticated right after signing up, enhancing security. * **Enable and Disable Sync via UI**\ Users can now control sync processes directly from the UI, giving flexibility to manage syncs as needed. * **Disable Catalog Validation for Data Models**\ Catalog validation is now disabled for non-AI data models, improving compatibility and accuracy. * **Model Query Preview API Error Handling**\ Added try-catch blocks to the model query preview API call, providing better error management and debugging. * **Fixed Sync Mapping for Model Column Values**\ Corrected an issue in sync mapping to ensure accurate model column value assignments. * **Test Connection Text**\ Fixed display issues with the "Test Connection" text, making it clearer and more user-friendly. * **Enable Catalog Validation Only for AI Models**\ Ensures that catalog validation is applied exclusively to AI models, maintaining model integrity. * **Disable Catalog Validation for Data Models**\ Disables catalog validation for non-AI data models to improve compatibility. * **AIML Source Schema Components**\ Refined AI/ML source schema components, enhancing performance and readability in configurations. * **Setup Charting Library and Tailwind CSS**\ Tailwind CSS integration and charting library setup provide better styling and data visualization tools. * **Add Model Name in Data App Response**\ Model names are now included in data app responses, offering better clarity for users. * **Add Connector Icon in Data App Response**\ Connector icons are displayed within data app responses, making it easier to identify connections visually. * **Add Catalog Presence Validation for Models**\ Ensures that a catalog is present and validated for all applicable models. * **Validate Catalog for Query Source**\ Introduces validation for query source catalogs, enhancing data accuracy. * **Add Filtering Scope to Connectors**\ Allows for targeted filtering within connectors, simplifying the search for relevant connections. * **Common Elements for Sign Up & Sign In**\ Moved shared components for sign-up and sign-in into separate views to improve code organization. * **Updated Sync Records UX**\ Enhanced the user experience for sync records, providing a more intuitive interface. * **Setup Models Renamed to Define Setup**\ Updated terminology from "setup models" to "define setup" for clearer, more precise language. *** > For further details on any feature or update, check the detailed documentation or contact our support team. We’re here to help make your experience seamless! *** # Overview