Skunkworks 2022: A Week of Building for MongoDB Engineers

MongoDB’s 2022 internal Skunkworks hackathon wrapped up in July, and it was a bustling time of hacking, building, and developing. For MongoDB engineers, Skunkworks is a week of no meetings and no interviews—engineers can have fun and work alongside co-workers from different teams to build side projects, proof of concepts, or anything else imaginable.

Many companies do one- or two-day hackathons, maybe once per year. We do one-week hackathons about every eight to ten months. Why? Hackathons are an important part of our engineering culture and embody our values of “Think Big, Go Far” and “Build Together.” We find that dedicating an entire week to the hackathon leads to significant innovation.

With more than 115 projects submitted, this year’s Star Trek-themed event was one of the best and largest hackathons yet. Building from some of the newest MongoDB 6.0 features, let’s look at a few winning projects and the people behind them.

Charts Slack Integration

Team: James Wang, Ryan Nguyen, Andrew McMenemy, and Muthukrishnan Krishnamurthy

Image containing the Slack and MongoDB Charts logos

We work on the Charts team and genuinely love our product; plus, it’s always fun taking complete ownership of it for a week. Reporting is a big ticket item for any data visualization tool. It’s something we’ve always had in the back of our minds, and users seem really keen on it. Variations of this request have been asked on our User Voice page since 2019, along with a request for a Slack integration with Charts since 2020.

This feature is built on top of our Embedded Charts SDK. We updated the URL used to fetch an embedded Chart/Dashboard to take on a new query parameter for screen grabbing. This new URL will trigger an AWS Lambda instance we wrote for the project, which will make use of our embedding SDK to embed and take a screenshot of the chart.

Image displaying charts hourly customer interaction utilizing different tables

Skunkworks is our favorite time of the year. Learning software through projects is so much easier said than done, especially once you start a full-time career in software. MongoDB’s hackathon improves us as engineers, provides a break from the standard work week, and allows us to work on whatever we want, whether it’s a solo project or helping to make someone's dream a reality. It inspires us to always be looking for the next best idea, along with it being a great mental health week.

MongoDB Carbon Footprint Calculation

Team: Nellie Spektor, Maya Raman, Cathy Wang, Rohan Chhaya, and Tiffany Feng

Our team was inspired by a previous Skunkworks hackathon project focused on sustainability within MongoDB Atlas. We decided to do a deep dive on carbon footprint measurement within MongoDB. We began by investigating the carbon efficiency of various parts of MongoDB, from drivers to Atlas.

For testing Atlas, we first set up multiple clusters, each varying in either location, cluster size, and sharding status. Then, using an atlas-co2 calculator script that a MongoDB Developer Advocate made, we were able to test how much carbon each cluster was emitting. Finally, we were able to display our findings about different cluster sizes and cloud providers and their carbon footprints.

While investigating drivers, we tested 100 insert/find/update/delete operations on 7 of our drivers and calculated the time taken and the wattage used, which was used to calculate carbon footprint and rank the drivers in terms of efficiency.

The biggest takeaway for us was the sheer difference in carbon emissions that a simple choice can make. For example, using the Rust driver instead of Java uses 144 times less electricity and therefore emissions. Simply shifting your cluster from one cloud provider in the Virginia region to a different cloud provider in the Iowa region saves over 3kg of carbon a week.

MongoDB is collaborating with a third-party vendor to get more robust carbon emission calculations, while tangentially understanding how we can provide a more efficient, sustainable product. Efforts like these help us to better refine our overall corporate emissions calculations.

The Skunkworks hackathon is a great initiative that shows engineers how much MongoDB values creativity and personal growth. It's amazing that we are actually encouraged to put aside our regular work and try out anything we’d like. While some people work on personal development projects, other people take the opportunity to tackle some tech debt or explore new features without the constraints of the normal product development process. Furthermore, it allows us to integrate our outside interests into our work, which makes us even more passionate and motivated.

Atlas Static Site Search

Team: Ben Perlmutter, Joon Young Lee, Shibi Balamurugan, Marcus Eagan, and Nick Larew

Our project was inspired by Algolia DocSearch. They’ve done a great job making it super easy to add search to a website. We wanted to make something similar with the MongoDB Atlas developer data platform, and we knew Atlas had the tools we needed to make this possible. It was just a question of writing some code to connect these services and creating a streamlined developer experience.

We called the project Atlas Static Site Search, and it has the following distinct components:

  • A website scraper that pulls site data and adds it to MongoDB built using Atlas Triggers.

  • A search index built with Atlas Search using the site data.

  • An Atlas Function that queries Atlas Search.

  • A React component that you can add to a website that uses the Realm Web SDK to call the Atlas Function that performs search.

  • A CLI that you can use to set up the whole backend (site scraper, search index, and search query function) with one command.

Image displaying the Atlas Static Site Search Architecture

The biggest challenge was getting all the different cloud services to work together well. Since it is a one week hackathon, we didn’t have time to write proper integration tests to validate that things were working as expected before deploying them. There was a lot of deploying code, praying it would work, it not working, and hotfixing.

The Skunkworks hackathon is one of our favorite parts of working at MongoDB. Whether you are an intern or an experienced engineer, you’re given the freedom to work on a passion project or learn something new. It speaks to the respect that MongoDB leadership has for the engineering and product teams. A lot of great ideas and innovative products have come out of hackathons in the past. We’ll see what happens with Atlas Static Site Search!

Simulating Common Customer Workloads

Team: Xiaochen Wu, Kyle Suarez, and Nishith Atreya

Our team recognized how replicating customer workloads has a myriad of benefits and can directly or indirectly help build a better testing environment, empower our support team, and identify potential improvement opportunities in our own product portfolio.

We tried to replicate two different workloads—one transactional and one in-app analytical. After identifying these workloads, we brainstormed how to use MongoDB features and products to support them. Then, we identified important characteristics of each workload and began replicating them using available datasets. Following this, we monitored how each workload performed in the MongoDB platform and collected insights and recommendations for our internal teams.

One of the biggest challenges was trying to figure out the important characteristics of each workload we were trying to replicate. For example, it was more difficult than expected to think of the most commonly used queries, search, and recommendation patterns that would appropriately represent the transactional customer workload.

After the completion of our project, we created a recommendation for MongoDB to build a workload suite consisting of workloads that cover a variety of customer industries. This would allow our engineering organization to test major upgrades, perform product research, and identify improvement opportunities in our platform.

$semanticSearch Aggregation Stage

Team: Thomas Rueckstiess and Steve Liu

At MongoDB Labs, we're always exploring how new technologies can be integrated with MongoDB. We read a paper that was published by a few researchers from Meta describing a novel architecture for semantic search and thought this could be a cool week-long project. We made four key changes:

  1. Built a web service that was the API interface to the model

  2. Introduced an aggregation pipeline called $semanticSearch that communicated with the API

  3. Deployed the web service on AWS

  4. Built a Star Trek themed front end using React

The Skunkworks hackathon helps us explore the creativity of MongoDB engineers. Every project submitted looked well polished and innovative. We walked away inspired by the talent that's evident in the business. Hackathons provide a creative outlet for engineers away from the day-to-day tickets and helps build a meritocratic culture where any project can receive recognition and reward.

Join us for the next hackathon: We’re actively hiring and looking for more talented, creative, and passionate engineers who want to build the next generation of MongoDB products and features!

← Previous

Built With MongoDB: Thunkable Brings the Power of App Development to Non-Developers

Great ideas can come from anywhere. But, if you have a great idea for a mobile app, you won't get far without developer talent. In today's app market, there are so many ideas in various stages of development that developer talent has become scarce and costly. To fill the gap, low-code and no-code solutions have emerged. Thunkable is a no-code platform that makes it easy to build custom native mobile apps without any advanced software engineering knowledge or certifications. The platform has seen tremendous growth, recently expanding to more than three million users. MongoDB has been pivotal to that growth, enabling the company to offer its services at scale without having to worry about managing the database. Thunkable is also part of the MongoDB for Startups program, which has helped the company solve some of the technical hurdles involved with scaling to millions of users. App creation for everyone Just because you don't have a computer science degree doesn't mean you can't come up with a great idea for an app. Thunkable co-founder and CTO, Wei Li, says the company is on a mission to democratize mobile app development. "We are currently empowering more than three million users across the globe that can come to our platform to build and publish their apps and do it without writing a single piece of code," Li says. Thunkable uses a simple drag-and-drop design canvas and powerful logic blocks to give innovators the tools they need to breathe life into their app designs. "It's very exciting seeing people from all different backgrounds trying to build solutions," says Jose Dominguez, engineer at Thunkable. "The part that I'm most proud about," says Li, "is our global users. Every day, I hear wonderful stories. For example, we recently have had people using our platform to build mobile apps to coordinate relief efforts in Ukraine. And since the pandemic, we have seen more users coming to our platform to build an app that addresses their needs for work, family, and community." Mobile phones are a transformational technology, but at the same time, there's untapped potential waiting to be exploited. Li believes Thunkable can help unleash the latent power lurking in our back pockets. "There are so many needs people could solve by using their smartphones," Li says. "But because they cannot program their phone, they become passive consumers. We want to empower them to become active creators." Modern app development with MongoDB Thunkable chose MongoDB early on because it mapped to its existing architecture; it stuck with MongoDB because it scaled when it became critical to do so. "We decided to build with MongoDB because it fits our data very naturally," Dominguez says. "We abstract our users' apps as documents, so it's a natural fit. We do a lot of writes so we needed a system to handle those kinds of loads and MongoDB was the perfect fit." Like a lot of startups, Thunkable has had to figure out how to achieve its goals with limited resources. Its engineering team consists of about four to six developers. "The engineering team has always been focused on building the product," Dominguez says. "So not having to worry about the database was a great win for us. It allowed us to iterate very fast and build new versions of the platform without having to worry about scaling the database or backups." After scaling to three million users, the Thunkable engineering team needed to rethink some of its design decisions about data. So, they talked with MongoDB engineers, courtesy of the startup program. Since then, their data storage needs have decreased while performance has improved. "Our partnership with MongoDB has been fundamental to our growth," Dominguez says. Li concurs: "As we scale, supporting more enterprise customers, we don't have to worry about database management issues. We know MongoDB will do a great job helping us as we scale." Building better together In addition to free MongoDB Atlas credits and one-on-one technical support, participants in the MongoDB for Startups program enjoy co-marketing opportunities and access to our partner network. "It's wonderful to be part of the MongoDB for Startups program," Li says. "We have all the support we need, from database management, MongoDB upgrades, and maintenance. You just go to one portal, one website. It's wonderful." MongoDB technical support has also been a life-saver for Thunkable, says Dominguez. "We had an initial call with an engineer and went through our logs. The engineer spotted a few things that we could fix right away. We were having issues with our op log — the tool that MongoDB uses to replicate to other servers — and the engineers helped get us out of those issues." The MongoDB dashboard is another tool Dominguez cites as being especially helpful. "For us, being able to log in to the dashboard and have all the functionality already there is a life saver," he says. "Not only being able to monitor the cluster, but digging into our collections and seeing if our indices are performing properly gives us tremendous insight. The integration of MongoDB and Google Cloud makes our infrastructure much easier to maintain." Future integrations Although they've scaled to millions of users, the team at Thunkable isn't finished. "This upcoming year will be a big one for Thunkable," Li says. "We're going to use our recent funding to scale up our support for enterprise, mainly for team support, collaboration, and enterprise design integration. For example, we recently integrated with Figma so you can import your Figma design into the Thunkable platform and have a functional app." "We're also currently hiring," Li adds. "We're planning to double our team within the next six months." To learn more about Thunkable or if you want to get started building your own mobile apps, check out Thunkable to get started for free . Are you part of a startup and interested in joining the MongoDB for Startups program? Apply now .

October 12, 2022
Next →

Unmasking Deception: Harnessing the Power of MongoDB Atlas and Amazon SageMaker Canvas for Fraud Detection

Financial services organizations face growing risks from cybercriminals. High-profile hacks and fraudulent transactions undermine faith in the industry. As technology evolves, so do the techniques employed by these perpetrators, making the battle against fraud a perpetual challenge. Existing fraud detection systems often grapple with a critical limitation: relying on stale data. In a fast-paced and ever-evolving landscape, relying solely on historical information is akin to driving by looking into the rearview mirror. Cybercriminals continuously adapt their tactics, forcing financial institutions to stay one step ahead. The newest tactics often can be seen in the data. That's where the power of operational data comes into play. By harnessing real-time data, fraud detection models can be trained on the most accurate and relevant clues available. MongoDB Atlas, a highly scalable and flexible developer data platform, coupled with Amazon SageMaker Canvas, an advanced machine learning tool, presents a groundbreaking opportunity to revolutionize fraud detection. By leveraging operational data, this synergy holds the key to proactively identifying and combating fraudulent activities, enabling financial institutions to safeguard their systems and protect their customers in an increasingly treacherous digital landscape. MongoDB Atlas MongoDB Atlas , the developer data platform is an integrated suite of data services centered around a cloud database designed to accelerate and simplify how developers build with data. MongoDB Atlas's document-oriented architecture is a game-changer for financial services organizations. Its ability to handle massive amounts of data in a flexible schema empowers financial institutions to effortlessly capture, store, and process high-volume transactional data in real-time. This means that every transaction, every interaction, and every piece of operational data can be seamlessly integrated into the fraud detection pipeline, ensuring that the models are continuously trained on the most current and relevant information available. With MongoDB Atlas, financial institutions gain an unrivaled advantage in their fight against fraud, unleashing the full potential of operational data to create a robust and proactive defense system. Amazon SageMaker Canvas Amazon SageMaker Canvas revolutionizes the way business analysts leverage AI/ML solutions by offering a powerful no-code platform. Traditionally, implementing AI/ML models required specialized technical expertise, making it inaccessible for many business analysts. However, SageMaker Canvas eliminates this barrier by providing a visual point-and-click interface to generate accurate ML predictions for classification, regression, forecasting, natural language processing (NLP), and computer vision (CV). SageMaker Canvas empowers business analysts to unlock valuable insights, make data-driven decisions, and harness the power of AI without being hindered by technical complexities. It boosts collaboration between business analysts and data scientists by sharing, reviewing, and updating ML models across tools. It brings the realm of AI/ML within reach, allowing analysts to explore new frontiers and drive innovation within their organizations. Reference Architecture The above reference architecture includes an end-to-end solution for detecting different types of fraud in the banking sector, including card fraud detection, identity theft detection, account takeover detection, money laundering detection, consumer fraud detection, insider fraud detection and mobile banking fraud detection to name a few. The architecture diagram shown here illustrates model training and near real-time inference. The operational data stored in MongoDB Atlas is written to the Amazon S3 bucket using the Triggers feature in Atlas Application Services. Thus stored, data is used to create and train the model in Amazon SageMaker Canvas. The SageMaker Canvas stores the metadata for the model in the S3 bucket and exposes the model endpoint for inference. For step-by-step instructions on how to build the fraud detections solution mentioned above with MongoDB Atlas and Amazon SageMaker Canvas, read our tutorial .

June 21, 2023