Fueling Pricing Strategies with MongoDB and Databricks

Deploying real-time analytics solutions with the right tech stack can have transformative benefits. Retailers want to grow their brand name or improve customer experience with value based pricing, whilst remaining competitive and cost effective.

Despite their ambition to become “data driven” operations, companies often fail in their endeavors, at the core of this is the struggle to do real-time analytics.

We will explore the architecture in Figure 1 and discuss the advantages of integrating MongoDB Atlas and Databricks as a perfect pairing for retail pricing strategies using real time AI. The solution we’ll describe integrates concepts from Event Driven architecture in the data generation and ingestion side, real time analytics orchestration, machine learning and microservices. Let’s get started!

Figure 1:  Overview of a dynamic pricing solution architecture

Reduce friction with flexibility

The pricing data complexity for a retailer with a diverse product line increases due to factors like seasonal sales, global expansion, and new product introductions. Tracking and analyzing historical prices and product offerings become more challenging as they change throughout the year.

Analytics solutions built around event driven architectures try to explain what is happening in a specific system or solution based on any significant occurrence such as user actions, data updates, system alerts, or sensor readings.

Deciding which occurrences are crucial to understand your customers and instrument your business model around that is when things start to become more intricate. Specially when trying to instrument your data models using traditional relational database management systems and its disadvantage when it comes to pairing that data structure with object oriented applications.

The inability of a retailer to adapt it’s data model to the customer behavior quickly translates into friction and a weaker presence in the market, for example poor pricing strategies compared to competitors because they lack information of historic price points and how they vary between products.

Figure 2:  An inflexible data model is a road block for innovation

That friction is contagious throughout the whole value chain of an organization, affecting the semantic layer of business (a bridge between the technical data structures and business user understanding), generating data inconsistencies, and reducing time to insight.

The capacity of your conceptual data model to adapt to an ever changing customer behavior helps reduce that friction significantly as its flexibility allows for a more intuitive data modeling of real world events.

For your real time pricing strategies challenges, MongoDB Atlas document model with its embedding and extended reference capabilities becomes the perfect tool for the job, as it allows for faster feature development and stronger test driven growth and talent retention as a consequence.

In combination with it’s high performance queries and horizontal scalability the solution becomes robust as it will handle the high throughput of clickstreams on your ecommerce applications and yet be able to respond real time data driven decision making features.

Its ease of integration with other platforms thanks to strong API capabilities and drivers make it the perfect solution on top of which to build your business operational and intelligence layers as you’ll avoid vendor lock-in and data scientists can easily leverage AI frameworks to work with fresh data.

It’s distributed by default principle, plus following best practices principles guarantee your operational data layer will handle the workload needed. As the AI algorithms analyze vast amounts of historical and real-time data to make pricing decisions, having a distributed platform like MongoDB enables efficient storage, processing, and retrieval of data across multiple nodes.

From what? To why? The intelligence layer

To unlock meaningful market growth and achieve it at scale your analytics need to evolve from just understanding what is happening by querying and analyzing historical data, to understand why the events measured with your operational data layer are happening and even further try to forecast them.

For a pricing solution, retailers would need to gather historical pricing points data for their product lines and shape through ETL (Earn, Transform, Load) pipelines to feed machine learning algorithms. This process is often complicated and brittle using the traditional data warehousing approach, often incurring in data duplication making it difficult and costly to manage.

Figure 3:  Reduced friction thanks to seamless integration of the different data layers

The advantage of using MongoDB Atlas as your operational data layer solution, is that through its Aggregation Pipelines you can shape your data in any way you need and then through MongoDB App Services, you can instrument Triggers and Functions to simplify this process and then consume that data in Databricks by leveraging the MongoDB Atlas via Spark connector.

Databricks provides you with a streamlined way of working with your models, by writing python code on hosted clusters notebooks. You can leverage its MLFlow integration to be able to register experiments which then can be turned into deployed models over an endpoint.

So transforming your data and integrating your operational layer, through connectors and API calls as triggers and functions, with your intelligence layer for machine learning and AI, you can easily build a pricing solution that will be able to generate market growth for your organization from its core through a semantic layer acting as a bridge between the technical aspects of data storage and the business requirements of data analysis.

Uncover new growth opportunities

Designing a real time analytics solution with MongoDB Atlas and Databricks is not only the fastest way to unlock your team's capabilities to devise pricing strategies, it also sets the cornerstone to build automated rules for more complex solutions.

Other ways of automating your retail application with AI driven insight could include: optimizing your marketing mix budget by each product price elasticity, adding another analytical layer of customer segmentation data to achieve targeted dynamic pricing, or optimizing your supply chain with sales forecasting in real time.

By taking advantage of MongoDB Charts or the MongoDB BI Connector, you can fuel your business dashboards, making that semantic layer of the business model the central point for your teams alignment.

Foundations for growth

Modern ecommerce sites unleash the power of real time analytics and automation to create better experiences for customers and a more profound approach to customer analytics by unblocking the power of machine learning to discover trends in behavioral data, effectively turning companies into automated growth machines.

If you want to discover how to build a simple dynamic pricing solution integrating MongoDB Atlas and Databricks make sure to read this guide.

← Previous

Modernize Your Factory Operations: Build a Virtual Factory with MongoDB Atlas in 5 Simple Steps

Thank you to Karolina Ruiz Rojelj for her contributions to this post. Virtual factories are revolutionizing the manufacturing landscape. Coined as the "Revolution in factory planning" by BMW Group at NVIDIA, this cutting-edge approach is transforming the way companies like BMW and Hyundai operate, thanks to groundbreaking partnerships with technology companies such as NVIDIA and Unity. At the heart of this revolution lies the concept of virtual factories , computer-based replicas of real-world manufacturing facilities. These virtual factories accurately mimic the characteristics and intricacies of physical factories, making them a powerful tool for manufacturers to optimize their operations. By leveraging AI, they unlock a whole new world of possibilities, revolutionizing the manufacturing landscape, paving the way for improved productivity, cost savings, and innovation. In this blog we will explore the benefits of virtual factories and guide you through the process of building your own virtual factory using MongoDB Atlas. Let’s dive in! Unlocking digital transformation The digitalization of the manufacturing industry has given rise to the development of smart factories. These advanced factories incorporate IoT sensors into their machinery and equipment, allowing workers to gather data-driven insights on their manufacturing processes. However, the evolution does not stop at smart factories automating and optimizing physical production. The emergence of virtual factories introduces simulation capabilities and remote monitoring, leading to the creation of factory digital twins, as depicted in Figure 1. By bridging the concepts of smart and virtual factories, manufacturers can unlock greater levels of efficiency, productivity, flexibility, and innovation. Figure 1:  From smart factory to virtual factory Leveraging virtual factories in manufacturing organizations provides many benefits including: Optimization of production processes and identification of inefficiencies. This can lead to increased efficiency, reduced waste, and improved quality. Aiding quality control by contextualizing sensor data with the manufacturing process. This allows analysis of quality issues and implementation of necessary control measures while dealing with complex production processes. Simulating manufacturing processes and testing new products or ideas without the need for physical prototypes or real-world production facilities. This significantly reduces costs associated with research and development and minimizes the risk of product failure. However, setting up a virtual factory for complex manufacturing is difficult. Challenges include managing system overload, handling vast amounts of data from physical factories, and creating accurate visualizations. The virtual factory must also adapt to changes in the physical factory over time. Given these challenges, having a data platform that can contextualize all the data coming in from the physical factory and then feed that to the virtual factory and vice versa is crucial. And that is where MongoDB Atlas , our developer data platform, comes in, providing synchronization capabilities between physical and virtual worlds, enabling flexible data modeling and providing access to the data via a unified query interface as seen in Figure 2. Figure 2:  MongoDB Atlas as the Data Platform between physical and virtual Factories Now that we’ve discussed the benefits and the challenges of building virtual factories, let’s unpack how simple it is to build a virtual factory with MongoDB Atlas. How to build a virtual factory MongoDB Atlas 1. Define the business requirements The first step of the process is to define the business requirements for the virtual factory. Our team at MongoDB uses a smart factory model from Fischertechnik to demonstrate how easily MongoDB can be integrated to solve the digital transformation challenges of IIoT in manufacturing. This testbed serves as our foundational physical factory and the starting point of this project. Figure 3:  The smart factory testbed We defined our set of business requirements as the following: Implement a virtual run of the physical factory to identify layout and process optimizations. Provide real-time visibility of the physical factory conditions such as inventory for process improvements. This last requirement is critical; while standalone simulation models of factories can be useful, they typically do not take into account the real-time data from the physical factory. By connecting the physical and virtual factories, a digital twin can be created that takes into account the actual performance of the physical factory in real-time. This enables more accurate predictions of the factory's performance, which improves decision-making, process optimization, and enables remote monitoring and control, reducing downtime and improving response times. 2. Create a 3D model Based on the previous business requirements, we created a 3D-model of the factory in a widely used game engine, Unity . This virtual model can be visualized using a computer, tablet or any virtual reality headset. Figure 4:  3D-model of the smart factory in Unity Additionally, we also added four different buttons (red, white, blue, and “stop”) which enables users to submit production orders to the physical factory or stop the process altogether. 3. Connect the physical and virtual factories Once we created the 3D model, we connected the physical and virtual factories via MongoDB Atlas. Let’s start with our virtual factory software application. Regardless of where you deploy it, be it a headset or a tablet, you can use Realm by MongoDB to present data locally inside Unity and then synchronize it with MongoDB Atlas as the central data layer. Allowing us to have embedded databases where there's resource constrainment and MongoDB Atlas as a powerful and scalable cloud backend technology. And lastly, to ensure data synchronization and communication between these two components, we leveraged MongoDB Atlas Device Sync , providing bi-directional synchronization mechanism and network handling. Now that we have our virtual factory set-up, let’s have a look at our physical one. In a real manufacturing environment, many of the shopfloor connectivity systems can connect to MongoDB Atlas and for those who don't natively, it is very straightforward to build a connector. At the shopfloor layer you can have MongoDB set up so that you can analyze and visualize your data locally and set up materialized views. On the cloud layer, you can push data directly to MongoDB Atlas or use our Cluster-to-Cluster Sync functionality. A single IoT device, by itself, does not generate much data. But as the amount of devices grows, so does the volume of machine-generated data and therefore the complexity of the data storage architecture required to support it. The data storage layer is often one of the primary causes of performance problems as an application scales. A well-designed data storage architecture is a crucial component in any IoT platform. In our project, we have integrated AWS IoT Core to subscribe to MQTT messages from the physical factory. Once these messages are received and filtered, they are transmitted to MongoDB Atlas via an HTTP endpoint. The HTTP endpoint then triggers a function which stores the messages in the corresponding collection based on their source (e.g., messages from the camera are stored in the camera collection). With MongoDB Atlas, as your data grows you can archive it using our Atlas Online Archive functionality. Figure 5:  Virtual and physical factories data flow In figure 5, we can see everything we’ve put together so far, on the left hand side we have our virtual factory where users can place an order. The order information is stored inside Realm, synced with MongoDB Atlas using Atlas Device Sync and sent to the physical factory using Atlas Triggers . On the other hand, the physical factory sends out sensor data and event information about the physical movement of items within the factory. MongoDB Atlas provides the full data platform experience for connecting both physical and virtual worlds! 4. Data modeling Now that the connectivity has been established, let's look at modeling this data that is coming in. As you may know, any piece of data that can be represented in JSON can be natively stored in and easily retrieved from MongoDB. The MongoDB drivers take care of converting the data to BSON (binary JSON) and back when querying the database. Furthermore, you can use documents to model data in any way you need, whether it is key value pairs, time series data or event data. On the topic of time series data, MongoDB Time Series allows you to automatically store time series data in a highly optimized and compressed format, reducing customer storage footprint, as well as achieving greater query performance at scale. Figure 5:  Virtual and physical factories sample data It really is as simple as it looks, and the best part is that we are doing all of this inside MongoDB Atlas making a direct impact on developer productivity. 5. Enable computer vision for real-time inventory Once we have the data modeled and connectivity established, our last step is to run event-driven analytics on top of our developer data platform. We used computer vision and AI to analyze the inventory status in the physical factory and then pushed notifications to the virtual one. If the user tries to order a piece in the virtual factory that is not in stock, they will immediately get a notification from the physical factory. All this is made possible using MongoDB Atlas and its connectors to various AI platforms If you want to learn more, stay tuned for part 2 of this blog series where we’ll dive deep into the technical considerations of this last step. Conclusion By investing in a virtual factory, companies can optimize production processes, strengthen quality control, and perform cost-effective testing, ultimately improving efficiency and innovation in manufacturing operations. MongoDB, with its comprehensive features and functionality that cover the entire lifecycle of manufacturing data, is well-positioned to implement virtual factory capabilities for the manufacturing industry. These capabilities allow MongoDB to be in a unique position to fast-track the digital transformation journey of manufacturers. Learn more: MongoDB & IIoT: A 4-Step Data Integration Manufacturing at Scale: MongoDB & IIoT Manufacturing with MongoDB

June 20, 2023
Next →

Unmasking Deception: Harnessing the Power of MongoDB Atlas and Amazon SageMaker Canvas for Fraud Detection

Financial services organizations face growing risks from cybercriminals. High-profile hacks and fraudulent transactions undermine faith in the industry. As technology evolves, so do the techniques employed by these perpetrators, making the battle against fraud a perpetual challenge. Existing fraud detection systems often grapple with a critical limitation: relying on stale data. In a fast-paced and ever-evolving landscape, relying solely on historical information is akin to driving by looking into the rearview mirror. Cybercriminals continuously adapt their tactics, forcing financial institutions to stay one step ahead. The newest tactics often can be seen in the data. That's where the power of operational data comes into play. By harnessing real-time data, fraud detection models can be trained on the most accurate and relevant clues available. MongoDB Atlas, a highly scalable and flexible developer data platform, coupled with Amazon SageMaker Canvas, an advanced machine learning tool, presents a groundbreaking opportunity to revolutionize fraud detection. By leveraging operational data, this synergy holds the key to proactively identifying and combating fraudulent activities, enabling financial institutions to safeguard their systems and protect their customers in an increasingly treacherous digital landscape. MongoDB Atlas MongoDB Atlas , the developer data platform is an integrated suite of data services centered around a cloud database designed to accelerate and simplify how developers build with data. MongoDB Atlas's document-oriented architecture is a game-changer for financial services organizations. Its ability to handle massive amounts of data in a flexible schema empowers financial institutions to effortlessly capture, store, and process high-volume transactional data in real-time. This means that every transaction, every interaction, and every piece of operational data can be seamlessly integrated into the fraud detection pipeline, ensuring that the models are continuously trained on the most current and relevant information available. With MongoDB Atlas, financial institutions gain an unrivaled advantage in their fight against fraud, unleashing the full potential of operational data to create a robust and proactive defense system. Amazon SageMaker Canvas Amazon SageMaker Canvas revolutionizes the way business analysts leverage AI/ML solutions by offering a powerful no-code platform. Traditionally, implementing AI/ML models required specialized technical expertise, making it inaccessible for many business analysts. However, SageMaker Canvas eliminates this barrier by providing a visual point-and-click interface to generate accurate ML predictions for classification, regression, forecasting, natural language processing (NLP), and computer vision (CV). SageMaker Canvas empowers business analysts to unlock valuable insights, make data-driven decisions, and harness the power of AI without being hindered by technical complexities. It boosts collaboration between business analysts and data scientists by sharing, reviewing, and updating ML models across tools. It brings the realm of AI/ML within reach, allowing analysts to explore new frontiers and drive innovation within their organizations. Reference Architecture The above reference architecture includes an end-to-end solution for detecting different types of fraud in the banking sector, including card fraud detection, identity theft detection, account takeover detection, money laundering detection, consumer fraud detection, insider fraud detection and mobile banking fraud detection to name a few. The architecture diagram shown here illustrates model training and near real-time inference. The operational data stored in MongoDB Atlas is written to the Amazon S3 bucket using the Triggers feature in Atlas Application Services. Thus stored, data is used to create and train the model in Amazon SageMaker Canvas. The SageMaker Canvas stores the metadata for the model in the S3 bucket and exposes the model endpoint for inference. For step-by-step instructions on how to build the fraud detections solution mentioned above with MongoDB Atlas and Amazon SageMaker Canvas, read our tutorial .

June 21, 2023