Challenge
The client consolidates data from multiple external sources, including 3rd parties. The data is then matched, refined, and served back to their clients as a value-adding product. The client wanted to use Databricks to ensure that the data is served in an automated and compliant fashion.
Solution
We implemented Databricks to generate insights into the data and published these results in the Databricks data catalogue for the end clients. We then automated the solution and created some reusable functions to anonymise the PII data in the data.
Outcome
The work demonstrated the potential of using Databricks to bring a level of refinement to product design and development while introducing high levels of automation and strong compliance. The client successfully delivered to their end clients and now has a clear path to further development of a slick, commercial product.
