Skip to main content

loaders.sqlserver

download_sql_to_parquet​

def download_sql_to_parquet(connection_string: str,
                            query: str,
                            parquet_file_path: str,
                            bronze_file_path: str,
                            partition_column: Optional[str] = None)

Download data from SQL using a query and save it as a Parquet file using Spark.

Arguments:

  • connection_string - ODBC connection string with authentication
  • query - SQL query to execute
  • parquet_file_path - Path where the Parquet file will be saved
  • bronze_file_path - Path to the bronze layer file
  • partition_column - Optional column name to partition the parquet file by

list_sql_tables​

def list_sql_tables(connection_string: str, schema: str = None)

List all tables in the specified schema.

Arguments:

  • connection_string - ODBC connection string with authentication
  • schema - Optional schema name to filter

Returns:

List of table names

generate_table_query​

def generate_table_query(table_name: str,
                         columns: list = None,
                         where_clause: str = None)

Generate a SQL query for a table with optional column selection and where clause.

Arguments:

  • table_name - Full table name (schema.table)
  • columns - Optional list of columns to select
  • where_clause - Optional WHERE clause

Returns:

SQL query string