Developer of new data technologies

[DonnescoPablo]

Developer of new data technologies

During the final semester of my studies, I completed a six-month end-of-study internship at Capgemini Technology Services, focusing on the theme "Developer of New Data Technologies".

Capgemini

Capgemini is the leading French IT consulting company, founded in 1967 in Grenoble. Today, it is the largest IT services firm in France. As a member of the CAC40, Capgemini has more than 340,000 employees across 50+ countries worldwide, with a 2023 revenue estimated at €22.5 billion.

Internship Overview

One of the reasons I was drawn to Capgemini is the wide range of cutting-edge technologies it works with, and the opportunity to be involved in exciting and impactful projects. I’d like to share two key projects I worked on during my internship.

Google Cortex Framework

One of Capgemini's clients, an automotive company, historically used SAP ECC and S4/HANA for resource planning. They sought to migrate their data from SAP systems to Google Cloud, aiming to reduce costs while taking advantage of Google Cloud’s advanced analytics capabilities.

To achieve this, the team employed the Google Cortex Framework. This solution provides a reference architecture, predefined templates, and deployment accelerators, which significantly reduce both the time and complexity required for migration.

The framework supports various data sources, including SAP, Salesforce, Google Ads, TikTok, and Meta.

Below is an outline of the architecture we used to integrate SAP data:

1. Pre-Configuration: First, we define the source systems, tables to replicate, and the frequency of updates using configuration files.
2. Deployment Pipelines: Using the provided deployment accelerators, Cloud Build triggers pipelines that generate artifacts in Cloud Storage and replicate the data into BigQuery under a "raw dataset". Since data changes (inserts, updates, deletes) are recorded continuously with timestamps, additional steps are needed to ensure the most up-to-date view of the data.
3. Change Data Capture (CDC): Cloud Composer orchestrates workflows that process the raw data into a "CDC Dataset". After that, it applies various analytical transformations, and the results are stored in a "Reporting Dataset".
4. Data Visualization: The framework offers ready-made templates for Looker, making it easy to visualize the processed data. Other visualization tools like Microsoft Power BI can also be used, though no predefined templates are provided for them.

Key Components Explained:

• Cloud Build: This is a continuous integration tool that automates the building and deployment of applications, services, or pipelines. In this case, it's used to automate the data replication process and trigger the necessary pipelines for the migration.
• Cloud Storage: This is a highly durable and scalable object storage service on Google Cloud, which stores all generated artifacts, such as configuration files and temporary data, required during the data migration process.
• BigQuery: Google’s fully-managed data warehouse designed for large-scale data analysis. It holds the replicated data and provides the capability for fast querying and processing of large datasets.
• Cloud Composer: Based on Apache Airflow, Cloud Composer is a fully managed workflow orchestration service that automates the scheduling and monitoring of data workflows. It handles the data transformation and update processes after replication into BigQuery.
• Looker: A powerful data visualization tool from Google Cloud, which allows businesses to explore, analyze, and share real-time data insights.

Generative AI and RAG (Retrieval Augmented Generation)

Generative AI, particularly models like ChatGPT, has gained immense popularity. This field of AI enables models to learn from vast amounts of data and generate meaningful outputs, such as text, images, or videos.

Among the many advancements in Generative AI, Foundation Models stand out. These models are trained on extensive public datasets and serve as the basis for Large Language Models (LLM), like ChatGPT, which can generate human-like text.

However, companies often face challenges when trying to leverage generative AI while safeguarding their private data. While some organizations build or fine-tune their own models, this requires significant computational resources and technical expertise. An alternative solution is Retrieval Augmented Generation (RAG).

How RAG Works:

1. Company documents are first transformed into vector representations and stored in a vector database.
2. When a user asks a question via a chatbot, the system retrieves relevant documents from the vector database.
3. These documents are used to form a prompt for the LLM, instructing it to answer the user’s question based solely on the retrieved information.

For one of our clients in the food industry, we designed an architecture based on RAG to support their technical support team in resolving tickets. The proposed architecture included:

• A Python backend hosted on Azure Web App.
• A vector database using CosmosDB for MongoDB.
• OpenAI models like GPT-4 for the chat and text-embedding-ada-002 for text vectorization.

In addition to automatic document retrieval, users can manually upload documents, or the system can be connected to external tools such as ServiceNow and Jira for ticket management or SharePoint for document handling.

Looking Ahead

Capgemini offers a wide range of career paths, whether you're inclined towards technical roles, functional positions, or a mix of both. The company also provides comprehensive internal training programs to help employees advance and achieve their career goals.

As for myself, I’m pursuing a career as a Solutions Architect. This role involves designing technical solutions tailored to client needs, while collaborating with development teams to bring these solutions to life.