All the deployments are automated and Ansible plays a central role. With the increasing complexity of the code base, we needed a new system to address the limitations of Ansible and tackle new challenges. TDP is a fully open source big data platform based on the Hadoop ecosystem. Alliage provides support and professional services for TDP.
One of the limitations of Ansible is that scheduling is not easy. It is difficult to trigger a task at the end of another task or select different tasks based on specific needs. In TDP, the primary way of controlling deployment is through variables. However, defining and versioning variables in Ansible can complicate matters, as there are currently 22 different locations where variables can be defined. As the project grows in complexity, it becomes difficult to keep track of variable definitions and re-definitions, and sometimes it may lead to defining defaults that are outside our control. To overcome this, we decided to add a 23rd way of easily versioning and adding custom behavior. We developed the necessary tools to properly version these variables, which resulted in the creation of tdp-lib.
Tdp-lib is an SDK that allows compatible collections to define a Directed Acyclic Graph (DAG) that outlines the relationships between components and services, specifying the execution order of tasks. It also allows for the definition of variables per service and per component in YAML files. With these features, tdp-lib can restart the minimal number of components in the TDP stack to apply configuration changes. It’s important to note that tdp-lib does not replace Ansible but uses it internally to deploy TDP.
The TDP roadmap includes the development of a set of tools to help manage cluster deployment. The main features of tdp-lib and tdp-server have been implemented, while tdp-ui and tdp-cli are still in active development or not yet started.
In order to precisely define what TDP can and cannot do, we have established a terminology. The fundamental concept of TDP is an operation, which can be a service operation or a component operation. Service operations consist of a service name and an action name, whereas component operations consist of a service name, a component name, and an action name. For example, “zookeeper_server_install” is a component operation.
The DAG is defined using YAML files located in the tdp_lib_dag folder of a collection. Each file contains a list of dictionaries with keys such as name, noop (optional), and depends_on. Although it’s possible to define the entire DAG in one file, it’s generally preferred to separate them.
Using variables in Ansible can be tricky, so we created a special place called TDP Vars to define and manage variables. This location does not exist when cloning collections from GitHub and needs to be set up during installation. The most common approach is to copy the variables from the tdp_vars_defaults folder in the collection path to the inventory folder. With the library, there are initialization functions available to perform this step.
Collections in TDP are entities that contain three folders: tdp_lib_dag, tdp_vars_defaults, and playbooks. The playbooks folder contains all the operations available in the collection, including special actions that are outside the DAG. Multiple collections can be used simultaneously by tdp-lib, and specific rules and behaviors are defined for managing dependencies and overriding operations and variables.
TDP-lib is a crucial tool for managing a TDP cluster, although it’s not meant to be used directly by users. The TDP ecosystem is evolving rapidly, and contributions are welcome to enrich its features.