# SDK - Getting Started

Perform the following steps to get started with SDK:

Setup SDK in your system
Create a new Connector Project
Develop the Connector
Manage Profiles for the Connector
Manage Credentials for the Connector
Running the Connector
Build and Publish the Connector

# Setup SDK in your system

Here are the requirements for SDK installation and connector development:

Python version 3.12 and above
A valid, active user account registered with Purple Fabric platform. You must have the following policies assigned to your account:
- GenAI_User
Access for the CONNECTOR_REPO_USERNAME and CONNECTOR_REPO_PASSWORD credentials in order to download the SDK
- To request access for these credentials, please fill the below Google form. The credentials will be sent via e-mail
  https://docs.google.com/forms/d/e/1FAIpQLSfnAfI1ICpZa3AkfWkRs91W0JfC-Go5BvVn7l1jfn-3IznUVg/viewform?usp=sharing
A minimum disk space of 150 MB.

Follow the below steps to install the SDK:

Download the installer file for your respective Operating System and Shell:
- Linux / Mac: install.sh
- Windows (Command Prompt): install.cmd
- Windows (PowerShell): install.ps1
Execute the installer file according to your Operating System and the shell:
- Linux / Mac:
  - Make the installer executable using the command chmod +x install.sh
  - Setup the environment variables using the commands:
    export CONNECTOR_REPO_USERNAME=<provide username here> export CONNECTOR_REPO_PASSWORD=<provide password here>
  - Run the executable using bash install.sh (if you are using bash)
  - For any other shell, execute the install.sh as required by the shell
  - You may have to restart the terminal in order for changes to take effect
- Windows (Command Prompt):
  - Open a Command Prompt in the directory where install.cmd is located
  - Setup the environment variables using the commands:
    set CONNECTOR_REPO_USERNAME=<provide username here>
    set CONNECTOR_REPO_PASSWORD=<provide password here>
  - Execute the command: .\install.cmd
- Windows (PowerShell):
  - Open a PowerShell window in the directory where install.ps1 is located
  - Setup the environment variables using the commands:
    set CONNECTOR_REPO_USERNAME=<provide username here>
    set CONNECTOR_REPO_PASSWORD=<provide password here>
  - Execute the command .\install.ps1
Finally, verify whether the command pf-cli is now available by executing it:
pf-cli

Follow the below steps to uninstall the SDK:

Uninstalling SDK can be helpful in order to re-install the SDK with the proper configuration.
Execute the installer file according to your Operating System and the shell:
- Linux / Mac: uninstall.sh
- Windows (Command Prompt): uninstall.cmd
- Windows (PowerShell): uninstall.ps1
Follow the steps used in the previous section to run the executable file according to your Operating System and shell

Note:

The uninstaller scripts only uninstalls pf-cli command and all installed SDK versions
It does not remove any connector projects created by the pf-cli command
Nor does it remove any Purple Fabric profiles and credentials stored in the system

# Create a new Connector Project

Run the following command to create a new connector project:
pf-cli create --name <connector_name>
Alternatively, create the connector project in the current directory
pf-cli create —-current-dir

Note:
If pf-cli is being shown as an invalid command / not found, ensure that you have properly installed the SDK in the previous steps
A sample project will be generated as per the folder structure as shown in the right side

Once the project is created, install the dependencies in a new virtual environment, by following the below steps:

  cd <connector_name>
  python3.12 -m venv .venv
  source .venv/bin/activate
  pip install -e . --extra-index-url=https://${CONNECTOR_REPO_USERNAME}:${CONNECTOR_REPO_PASSWORD}@iainexus.intellectseeccloud.com/repository/idxplatform-connectors-py-hosted/simple --verbose

Project Structure of a sample connector / tool
└── hello-world-connector/
├── .config/
├── generated/
│ ├── __init__.py
│ └── models.py
├── app.py
├── __init__.py
├── pyproject.toml
├── schema.json
├── README.md
├── Dockerfile
├── .dockerignore
└── .gitignore

Connectors and Tools

During project creation, you can pass an additional argument called --asset-category that accepts either INTEGRATOR or TOOLS
Default asset category is INTEGRATOR
INTEGRATOR asset category represents a family of data connectors in Purple Fabric, while TOOLS asset category represents a family of data transformers in Purple Fabric
Please choose the asset category that best represents the nature of the project
Agent category can be updated in pyproject.toml before the first build operation is executed

Details of the files and folders

Name	File/Directory	Description
.config	Directory	profiles and credentials created locally (project level). It will be empty by default.
generated	Directory	Contains files that are generated by the SDK using commands, like models.py Note: Do not edit any files in this directory
app.py	File	The main file that contains the implementations of the features mentioned in schema.json
__init__.py	File	Standard Python file that denotes the current directory is a Python module.
pyproject.toml	File	Standard Python file that defines the project’s metadata and dependencies.
schema.json	File	A JSON file that contains the specifications of the project’s features and schemas.
Dockerfile	File	Specifies the project’s operating system level dependencies.
.gitignore, .dockerignore	File	Standard .ignore files used in respective contexts.
README.md	File	Contains the project’s documentation.

# Develop the Connector

Note:

To proceed with the below steps, a profile is required
Profiles can be created using the pf-cli

Specification → Schema

Update schema.json to define the interface for your connector. Specify the input schema and output schema for your connector inside it
Generate Pydantic models for your schema.json by executing the pf-cli generate models --profile <profile_name> command. Models will be generated inside the generated directory
Use your models inside app.py and build your business logic inside the run_<feature_name>() function

Custom Dependencies

To install custom dependencies:
- Update the dependencies section inside pyproject.toml
- Ensure that you are using the project’s virtual environment and the virtual environment is activated
- Run the below command to install the custom dependencies
```
pip install -e . --extra-index-url=https://${CONNECTOR_REPO_USERNAME}:${CONNECTOR_REPO_PASSWORD}@iainexus.intellectseeccloud.com/repository/idxplatform-connectors-py-hosted/simple --verbose  
```

Implementation → Run function
Definition

In the app.py file, you will find a run function with the following signature:
def run_<FEATURE_NAME>(inputs: <FEATURE_NAME>Inputs, context: Context) -> <FEATURE_NAME>Outputs: …
<FEATURE_NAME> should match at least one feature mentioned in schema.json file
Add your business logic inside the run function
You can use custom libraries inside the run function as long as you have mentioned them in pyproject.toml and installed them in your virtual environment
At least one run function is required to execute the project
In case there is a feature defined in schema.json, but not implemented in app.py, the SDK will raise an exception during build phase

Create a new feature
To generate a new feature and its implementation, execute the below command:

**pf-cli generate feature \--name \<feature\_name\> \--profile \<profile\_name\>**

This command does the following operations:

Adds the boilerplate feature definition, input schema and output schema components in schema.json for the feature <feature_name>
Generates models for the new input schema and output schema of the feature <feature_name>
Imports these models in app.py
Adds a new run function for the <feature_name> feature in app.py

Relationship of a feature between schema.json and app.py

The features section in schema.json represents the specification of an operation and its input and output schemas
The specified features are implemented as run functions inside app.py. The feature is mapped to the function name in the form run_<FEATURE_NAME>
- Example, if the feature download_files is defined in schema.json, the corresponding implementing function’s name should equal to run_download_files
The input schema and output schema defined for the feature in the schema.json will be mapped to the respective <FEATURE_NAME>Input and <FEATURE_NAME>Output Pydantic models.
- Example, if the feature download_files is defined in schema.json, the corresponding input model name should equal to DownloadFilesInput and the corresponding output model name should equal to DownloadFilesOutput

Multi-feature support

A project can support multiple features and its respective implementations in the form of run functions
This helps in reducing code duplication as well as improving code re-usability of shared logic

Logging

You can add log statements to your application by using the context.logger() function

  def run_<FEATURE_NAME>(inputs: <FEATURE_NAME>Inputs, context: Context) -> <FEATURE_NAME>Outputs:
      context.logger.info('Hello, World!')
      ...   

Outputs

You must always return an object of type <FEATURE_NAME>Outputs
You must use the <FEATURE_NAME>Outputs Pydantic model available in generated/models.py module def run_<FEATURE_NAME>(inputs: <FEATURE_NAME>Inputs, context: Context) -> <FEATURE_NAME>Outputs: ... return <FEATURE_NAME>Outputs(name='Hello, World!')

Exceptions

You must use AssetException to raise any exception from your connector

When building custom exceptions, always ensure that AssetException is inherited

  from pf_connectors_py_sdk.core.exceptions import AssetException
  # other imports...

  class InvalidNameException(AssetException):
  def __init__(self, message):
      super().__init__(message)

  def run_<FEATURE_NAME>(inputs: <FEATURE_NAME>Inputs, context: Context) -> <FEATURE_NAME>Outputs:
      if inputs.name is None or inputs.name.startswith('123'):
          raise InvalidNameException('Name is invalid')
      return <FEATURE_NAME>Outputs(name='Hello, World!')

Credentials
Adding Credentials to the Connector

This step is only needed for the connectors that require access to credentials

In the schema.json file, add the new input section for connection and update the data type as "connection"

  {
  "datatype": "connection",
  "mandatory": true,
  "hidden": false,
  "displayname": "Connection",
  "properties": {
      "label": "Connection",
      "options": [],
      "placeholder_text": "Select Connection",
      "component": "connectionSelect",
      "help_text": "Choose a connection",
      "parameter": true,
      "value_key": "connection",
      "display_order": 1,
      "option_label_key": "name",
      "option_value_key": "connection_id",
      "provider_name": "Service Name"
  }
  }

Generate the models:
pf-cli generate models --profile <profile>

Credential Providers

Purple Fabric Platform provides support to create credentials for specific service providers like AWS, Google Cloud, Atlassian, etc.
In order to consume a specific credential provider in your connector, please provide the name of the credential provider in the provider_name field
{ "datatype": "connection", "mandatory": true, "hidden": false, "displayname": "Connection", "properties": { // ...other properties "provider_name": "Amazon Web Service" } }
After this, please generate the models to make use of the provider specific Pydantic model
pf-cli generate models --profile <profile>

Supported Credential Providers The list of supported credential providers is available here - Flow Designer: Asset Schemas

Note: You can use Generic Service provider in case no other credential provider suits your requirements.

# Manage Profiles for the Connector

Profiles are used for communication with Purple Fabric Platform to execute various commands.
Keep the below details available to create a profile:
- Environment
- Tenant ID
- API Key
- Username
- Password
Create a profile using pf-cli profiles create
Profiles can be re-used across projects, if it is created in the global scope

Note:
- The profile details will be validated before its created to ensure the provided details are valid
- Profile validation will be performed for all the commands that require a profile argument
- If your profile details are valid, and yet face issues with the SDK commands, you might need to check the policies assigned to your profile by your administrator.

# Manage Credentials for the Connector

Credentials can be created for any credential provider supported by Purple Fabric Platform
To create a credential, use the below command:
pf-cli credentials create --profile <profile>
You will be prompted to choose a credential provider. You can use a Generic Service provider in case no other credential provider suits your requirements

Note:
- Credentials will be created at workspace level
- You will be prompted to select a workspace from the available workspaces in your account
- When creating a credential via CLI, it will be created in your local system as well as in the Purple Fabric Platform
Once a credential is created, you will receive a connection_id
Use this connection_id when executing the connector that uses a connection field
pf-cli run --feature <feature> --inputs connection=<connection_id>

# Running the Connector

Run the connector locally using the following commands:

Use run to verify app.py is running as expected
You can only run a single feature at a time

# Build and Publish the Connector

To build the connector, run the below command:
pf-cli build --profile <profile>

Note:

When building the connector, it is bound to a specific workspace
You will be prompted to select a workspace from the available workspaces in your account
Once a workspace is selected, any future commands executed will be done on the chosen workspace

During build, the following validations will be performed:

Validation of schema.json
Validation of the asset details provided in pyproject.toml
Validate whether run function with the proper signature is present in app.py for all the features specified in schema.json

To publish the connector, run the below command:
pf-cli deploy --profile <profile>

During publish, your code will be converted into a deployment artifact and sent to the deployment pipeline
To check the status of the deployment, run the below command:pf-cli status --profile <profile>