Skip to main content

loaders.pull_files

pull_files​

def pull_files(config_manager: ConfigManager,
               table_config: TableConfig) -> List[ObjectInfo]

Pulls files from a remote source to a local destination based on the configuration and connection type provided. It handles various connection types, retrieves the file details, processes them, and returns a list of ObjectInfo instances representing the pulled files.

Arguments:

  • config_manager - Manages configurations used throughout the application.
  • table_config - Contains specific information and settings for a particular table including connection type, source folder, and bronze folder.

Returns:

  • List[ObjectInfo] - A list containing ObjectInfo instances, each representing a pulled file with relevant details such as its bronze file name, path, and source log.

Raises:

  • Exception - If a connection type is not set on the object for "customperfile".
  • Exception - If an unsupported connection type is detected.