> For the complete documentation index, see [llms.txt](https://whiteklay.gitbook.io/virtual-data-assistant/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://whiteklay.gitbook.io/virtual-data-assistant/how-to-guides/data-catalog/data-source.md).
# Data Source
### What is a Data Source
A Datasource is an entity within the Virtual Data Assistant that serves as a container for a collection of metadata. Metadata refers to information about datasets, such as data source location, schema, data types, and other relevant properties. Essentially, a Datasource is like a virtual folder that groups related datasets together, making it easier for users to manage and access data efficiently.
### How to Create a new Data Source
Navigate to Datasources and click create to create a new data source
Select the connector( source type ) radio button as per the required data source
### Advance Properties
If you do not want all tables, then you can add table names in Include or Exclude boxes
Include: Fetch only for these table(s)
Exclude: Fetch all excluding mentioned table(s)
Connectors are modules or plugins that establish connections to specific data sources or databases. VDA offers multiple connectors to popular datasources e.g PostgreSQL, MySQL etc. When users select a connector and provide the necessary connection details, it establishes a link to the data source, allowing access to the data within that source.
#### Postgres
Postgres data source type allows below two methods :-
Detail: Select this parameter when you are connecting using standard username , password and port .
URL: Select this parameter when you have a custom URL connection string Below is an example of a connection string
postgresql://user:password\@hostname:port/database
#### Oracle
Description: Use this method when connecting to an Oracle database using standard credentials and connection parameters.
Parameters:
Role: The role assigned to the user (e.g., DBA, Developer).
Host: The hostname or IP address of the Oracle server.
User: The username for the Oracle database.
Password: The password for the Oracle database.
Schema: The schema within the Oracle database to which the user has access.
Service: The Oracle service name (e.g., ORCL).
URL Method:
Description: Use this method when you have a custom URL connection string for the Oracle database.
#### Kafka
Name:
Description: The name of the Kafka connection.
Example: Kafka\_Prod\_Cluster
URL :
Description: The base URL for the Kafka service.
Importance: Provides the endpoint for connecting to the Kafka service.
Example: kafka.example.com
Broker URL:
Description: A comma-separated list of host and port pairs that are the addresses of the Kafka brokers.
Importance: Specifies the Kafka brokers to connect to.
Example: kafka1.example.com:9092,kafka2.example.com:9092
User:
Description: The username for the Kafka connection.
Importance: Used for authenticating the user accessing the Kafka cluster.
Example: kafka\_user
Password:
Description: The password for the Kafka connection.
Importance: Secures the connection by authenticating the user.
Example: securepassword
Secure Connection:
Description: A checkbox option to enable a secure connection.
Importance: Ensures that the data transmitted between the client and Kafka brokers is encrypted.
Example: Checked or Unchecked
Included Tables:
Description: A list of specific tables/topics to be included in the data ingestion process.
Importance: Allows for targeted data ingestion, focusing on relevant tables/topics.
Example: topic1, topic2, topic3
Excluded Tables:
Description: A list of specific tables/topics to be excluded from the data ingestion process.
Importance: Prevents unnecessary or irrelevant data from being ingested.
Example: topic4, topic5
#### Hive
Detail Method
Name:
Description: The name of the Hive connection.
Importance: Identifies the specific Hive data source within the VDA
Example: Hive\_Prod\_Cluster
Metastore URL:
Description: The URL of the Hive Metastore service.
Importance: Provides the endpoint for connecting to the Hive Metastore, which manages metadata for Hive tables and databases.
Example: thrift://metastore.example.com:9083
Hive URL:
Description: The URL of the Hive server.
Importance: Provides the endpoint for connecting to the Hive server for executing queries and accessing data.
Example: jdbc:hive2://hive.example.com:10000/default
Database:
Description: The name of the specific database within the Hive server to connect to.
Importance: Specifies the target database for data operations.
Example: default
Included Tables:
Description: A list of specific tables to be included in the data ingestion process.
Importance: Allows for targeted data ingestion, focusing on relevant tables.
Example: table1, table2, table3
Excluded Tables:
Description: A list of specific tables to be excluded from the data ingestion process.
Importance: Prevents unnecessary or irrelevant data from being ingested.
Example: table4, table5
#### SQL Server
Detail Method
Name:
Description: The name of the SQL Server connection.
Importance: Identifies the specific SQL Server data source within the VDA
Example: SQLServer\_Prod
Host:
Description: The hostname or IP address of the SQL Server.
Importance: Specifies the server where the SQL Server is hosted.
Example: sqlserver.example.com
Port:
Description: The port number used to connect to the SQL Server.
Importance: Specifies the network port for the SQL Server connection.
Example: 1433
User:
Description: The username for the SQL Server database.
Importance: Used for authenticating the user accessing the SQL Server.
Example: db\_user
Password:
Description: The password for the SQL Server database.
Importance: Secures the connection by authenticating the user.
Example: securepassword
Database:
Description: The name of the specific database within the SQL Server to connect to.
Importance: Specifies the target database for data operations.
Example: mydatabase
Schema:
Description: The schema within the SQL Server database.
Importance: Defines the organizational structure of tables within the database.
Example: dbo
Included Tables:
Description: A list of specific tables to be included in the data ingestion process.
Importance: Allows for targeted data ingestion, focusing on relevant tables.
Example: table1, table2, table3
Excluded Tables:
Description: A list of specific tables to be excluded from the data ingestion process.
Importance: Prevents unnecessary or irrelevant data from being ingested.
Example: table4, table5
URL Method
Description: Use this method when you have a custom URL connection string for SQL Server, incorporating all necessary connection details.
jdbc:sqlserver://sqlserver.example.com:1433;databaseName=mydatabase;user=db\_user;password=securepassword;schema=dbo
#### My SQL
Detail Method
Name:
Description: The name of the MySQL connection.
Importance: Identifies the specific MySQL data source within the VDA
Example: MySQL\_Prod
Host:
Description: The hostname or IP address of the MySQL server.
Importance: Specifies the server where the MySQL database is hosted.
Example: mysql.example.com
Port:
Description: The port number used to connect to the MySQL server.
Importance: Specifies the network port for the MySQL connection.
Example: 3306
User:
Description: The username for the MySQL database.
Importance: Used for authenticating the user accessing the MySQL database.
Example: db\_user
Password:
Description: The password for the MySQL database.
Importance: Secures the connection by authenticating the user.
Example: securepassword
Database:
Description: The name of the specific database within the MySQL server to connect to.
Importance: Specifies the target database for data operations.
Example: mydatabase
Schema:
Description: The schema within the MySQL database.
Importance: Defines the organizational structure of tables within the database.
Example: public
Included Tables:
Description: A list of specific tables to be included in the data ingestion process.
Importance: Allows for targeted data ingestion, focusing on relevant tables.
Example: table1, table2, table3
Excluded Tables:
Description: A list of specific tables to be excluded from the data ingestion process.
Importance: Prevents unnecessary or irrelevant data from being ingested.
Example: table4, table5
URL Method
Description: Use this method when you have a custom URL connection string for MySQL, incorporating all necessary connection details.
jdbc:mysql://mysql.example.com:3306/mydatabase?user=db\_user\&password=securepassword
\
List of Available Connectors (Grouped by Category)
Database Connectors (54) Amazon Athena, Azure SQL, BigQuery, BigTable, BurstIQ, Cassandra, Clickhouse, Cockroach, Couchbase, Custom Database, Databricks, Datalake, Db2, DeltaLake, DomoDatabase, Doris, Dremio, Druid, DynamoDB, Epic, Exasol, Glue, Greenplum, Hive, Iceberg, Impala, Informix, MariaDB, Microsoft Access, Microsoft Fabric, MongoDB, MSSQL, MySQL, Oracle, PinotDB, Postgres, Presto, Redshift, SAS, SQLite, SSAS, Salesforce, SAP ERP, SAP HANA, ServiceNow, SingleStore, Snowflake, StarRocks, Synapse, Teradata, Timescale, Trino, Unity Catalog, Vertica
Dashboard Connectors (20) Custom Dashboard, Domo Dashboard, Grafana, Hex, Lightdash, Looker, Metabase, MicroStrategy, Mode, PowerBI, PowerBI Report Server, QlikCloud, QlikSense, QuickSight, Redash, Sigma, Ssrs, Superset, Tableau, ThoughtSpot
Pipeline Connectors (25) Airbyte, Airflow, Backend, Custom Pipeline, DBT Cloud, Dagster, Data Factory, Databricks Pipeline, Domo Pipeline, Fivetran, Flink, Glue Pipeline, Kafka Connect, Kinesis Firehose, Matillion, Microsoft Fabric Pipeline, Mulesoft, Nifi, OpenLineage, SSIS, Snowplow, Spark, Spline, Stitch, Wherescape
Messaging Connectors (5) Custom Messaging, Kafka, Kinesis, Pulsar, Redpanda
LLM (Large Language Model) Connectors (8) Anthropic, Azure OpenAI, Bedrock, Custom LLM, HuggingFace, Ollama, OpenAI, VertexAI
Storage Connectors (4) Adls, Custom Storage, GCS, S3
Search Connectors (3) Custom Search, Elasticsearch, OpenSearch
Drive Connectors (4) Custom Drive, Google Drive, SFTP, SharePoint
ML Model Connectors (5) Custom ML Model, Mlflow, SageMaker, Sklearn, VertexAI
Metadata Connectors (7) Alation, AlationSink, Amundsen, Atlas, Collibra, MetadataES, OpenMetadata
API & Security Connectors (2) REST API, Ranger (Security)
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://whiteklay.gitbook.io/virtual-data-assistant/how-to-guides/data-catalog/data-source.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.