Enabling smallcase Offers for the greater audience using Kafka

At smallcase, offers means more than giving discounts to a user, we provide offers to encourage the newer audience to try the platform and discover the possibilities of investing in smallcases.

Utilising offers to grow the number of users of the product isn't a new thing and a lot of startups are doing this in the industry. smallcase also provides offers for special occasions such as Diwali, New Year to a smaller number of people. But with the rate of growth we are having, that quickly became a bottleneck as activating offers was becoming quite a tedious task.

Activating offers on smallcase platform is now a piece of cake, anyone can do this with some few clicks.

This becomes possible because of the revamped Offers microservice, which decouples the application and redemption of an offer from the main platform services and provides a simple interface which can be used to roll out new offers.

Setbacks with the previous implementation

As said the previous implementations had some problems which arose due to the scale as well the sudden necessity of adding a wide range of offers to different cohorts.

NOTE: The term of applying an offer to the user means making the user eligible for an offer, whereas the redemption of the offer only happens if the user performs any transaction on the platform or app.

Let's discuss one by one, the issues which we were facing:

• Scope:

  The variety of offers was really small due to the supporting infrastructure. Offers were rolled out only on special occasions such as Diwali, but now offers service is used to power various types of offers.

• Complexity:

  The older offers were having a rather simpler logic and were given to the users more directly, for example, all the users who haven't invested yet were found and offers were given to them. That's it, that was the only scope which the older offers had.

• Dependency on Code:

  The redemption of an offer was done via an event which only checked that the offer is available for the user, then the code had some if conditions which applied the offer to the user if those conditions were passed.

  So for every new offer, either the offer should be similar to pass the written conditions or the code needed some changes to add new conditions for every new offer which will need deployments each time.

• Manual Work:

  Whenever an offer was to be introduced on the platform, a dedicated person with the backend team was assigned who ran queries to make a user eligible for an offer. The query included the criteria for matching the user which is eligible for the offer. This took time and manual intervention, resulting in a solution which is not scalable.

sc-khata to the rescue

sc-khata(khata is a hindi word meaning account) is a new service written with sole purpose of managing the smallcase ledger of all the user transactions on the smallcase platform, including all the applicable offers (moving the ledger to sc-khata is a future task, the ledger service is currently an old cron-job).

Khata is based on Event Driven Architecture, where it consumes the event produced by the smallcase API and Order Management System primarily, for the processing of the offer.

Most of the event driven services at smallcase use Kafka because of the advantages it provides:

• Fault tolerant: The inherent capability of Kafka to be resistant to node/machine failure within a cluster.
• Durability: The data/messages are persistent on disk, making it durable and messages are also replicated. Messages can be replayed if necessary, or in a time of down-time.
• High-throughput: Kafka is capable of handling high-velocity and high-volume data using not so large hardware. It is capable of supporting message throughput of thousands of messages per second.
• Low latency: Kafka is able to handle these messages with very low latency of the range of milliseconds, demanded by most of new use cases.

So, all these advantages made Kafka an easy choice to implement this service that works asynchronously parallel to the main platform services and completes the required job without a high latency value.

How sc-khata works?

1. An offer which is currently active in the smallcase eco-system exists.

2. If the user is eligible for the offer, the offer gets added to the user account for future use.

3. Whenever a user places an order, the offer gets redeemed (from the user's account) and the offer expires.

There were some decisions taken to solve the above said problems, let us talk about them one-by-one.

Shifting the offer logic to the database

Writing the offer logic in the code was not scalable and as already said, to add a new offer, making changes in code and redeployment doesn't makes sense. So now all the offers logic including eligibility, redemption as well as invalidation, is written in the database. This provides us to add and update offers on the fly. 🚀

But how do you write logic in the database you ask? Right? 🤯

Before explaining that, you need to know how an offer looks like to the system. An offer document is attached below (we use MongoDB which is a NoSQL DB, therefore data is stored similar to what is displayed).

Copycopy code to clipboard
1{
2  "status": "AVAILABLE",
3  "offerCode": "FREE_AWI",
4  "includedScids": ["SCAW_0001"],
5  "endDate": ISODate("2020-03-13T10:00:00.000Z"),
6  "startDate": ISODate("2020-02-23T18:30:00.000Z"),
7  "discount": {
8    "type": "percentage",
9    "value": 1.0
10  },
11  "eligibility": {
12    "model": "User",
13    "operation": "count",
14    "query": "{\"broker.userId\":\"<%=brokeruserId%>\",\"investedSmallcases\":[]}",
15    "type": "query",
16    "expectedResult": 1.0
17  },
18  "redemption": {
19    "model": "Order",
20    "operation": "count",
21    "query": "{\"brokeruserId\":\"<%=brokeruserId%>\", scid: \"SCAW_0001\", \"batchId\":\"<%=batchId%>\", \"date\":{\"$gte\":\"<%=startDate%>\"}",
22    "type": "query",
23    "expectedResult": 1.0
24  },
25  "invalidation": {
26    "model": "Order",
27    "operation": "count",
28    "query": "{\"brokeruserId\":\"<%=brokeruserId%>\",\"batchId\":\"<%=batchId%>\",\"date\":{\"$gte\":\"<%=startDate%>\"},\"buyAmount\":{\"$gt\":0},\"originalLabel\":\"BUY\"}",
29    "type": "query",
30    "expectedResult": 1.0
31  }
32}

Some keys are pretty clear like offerCode, startDate and endDate, and the others are explained below:

• status: It contains the status of the offer which can either be AVAILABLE, EXPIRED or USED.
• includedScids: This array contains all the smallcase ids on which the offer is applicable. Here the offer is only for All Weather Investing(AWI) smallcase.
• discount: Discount can be of 2 types, either a percentage value on the charged amount, or a flat fee discount.
• eligibility, redemption, invalidation: All these properties contains some specific keys, which are used in calculating whether or not to complete the respective step. Let's talk about this in detail.

Total control, including graphical

In addition to these above keys we have another key called meta which plays another significant and beautiful role (quite literally) 🎊🎉

Copycopy code to clipboard
1{
2  "meta": {
3    "cta": {
4      "primary": {
5        "text": "See smallcases",
6        "action": "DISCOVER",
7        "query": "scids=SCAW_0001&scids=SCSB_0001&scids=SCSB_0003&scids=SCSB_0004",
8        "path": "/discover/all"
9      },
10      "secondary": {
11        "text": "See all smallcases",
12        "action": "DISCOVER"
13      }
14    },
15    "texts": {
16      "tnc": "Ends 3:30 PM, 10th Jan 2020. Brokerage charges still apply",
17      "title": "A holiday gift for you",
18      "description": "Zero lifetime fees on All Weather Investing or any one of Smart Beta smallcases.",
19      "subtitle": "Buy now at no lifetime fees"
20    },
21    "images": {
22      "background": "https://assets.smallcase.com/images/offers/NEWYEAR_2020/background.png",
23      "foreground": "https://assets.smallcase.com/images/offers/NEWYEAR_2020/foreground.png"
24    },
25    "style": {
26      "backgroundGradient": {
27        "startColor": "#7332ff",
28        "endColor": "#7332ff"
29      },
30      "border": {
31        "color": "#fff"
32      }
33    }
34  }
35}

This object controls how the banner on the smallcase dashboard will look like. From the bold eye catching image to the vivid background gradient color, it provides all the basic info that an offer banner needs. Even the text and the action performed by the button on the banner can be configured with the properties here.

All this results to this beauty that you see on the smallcase dashboard

Executing the offers logic

Continuing the explanation of properties of the offer document.

We are using the queries in the doc (highlighted keys above) to query the DB again to find out whether or not the user is eligible for the offer and so on. Every necessary value required to query is provided in the offer document already:

• type: Currently the type can be query (like the database query), but for future it can consist of values like underscore where instead of querying to the database we use libraries like underscore for calculation.

• operation: This key is used when the type key equals query. It can have values such as aggregate, count which specifies the type of query which we want.

• model: Here we specify the collection that needs to be queried. Again only used in case type key equals query.

• expectedResult: Finally the calculated result is compared to this key. If both are equal then the operation is considered to be successful.

• query: The real brainstorming was done here, keeping a database query inside the database.

  To convert this non functioning string into a Mongo DB query, something clever was done.
  Nah! Just kidding we just used a simple function provided by underscore, template.
  We use this to interpolate <%= … %> values with the specific user's properties dynamically and then parsing it to JSON.
  This helps us create a query ready to be executed.

We use mongoose as the ODM, which let us use the above parsed JSON perfectly to query the database by putting all the values together.

Let's take an example of the eligibility key from the offer document above:

Copycopy code to clipboard
1const _ = require('underscore');
2const { query, model, operation } = offer.eligibility;
3
4// the query is parsed because the query string in
5// the offer document is an undercore string template where
6// the variables can be replaced with the provided value
7const parsedQuery = JSON.parse(
8  _.template(query)({
9    brokeruserId: event.data.brokeruserId,
10  })
11);
12// this gives us a parsed JSON query which can be further
13// passed on to the mongoose model for querying
14
15// models is an Object which contains the map of
16// the collection name to mongoose model of that collection
17const result = await models[model][operation](parsedQuery).exec();
18
19if (result === expectedResult) {
20  // do the needful
21} else {
22  // skip everything
23}

We saw the code, so now, let's process what is happening logically.
The query above checks the count of the user whose broker id was provided to verify if the user has 0 invested smallcases.
If yes, the query returns the count as 1 which equals to the expected result which further means that the user is qualified for the offer and the offer gets pushed to the user.
If the count came 0, maybe because the user was already invested, then the user wouldn't have got the offer.

But where and when does all of this happens afterall?

Khata has 3 modules. Eligibility, redemption and invalidation.
As you might have noticed each of these modules have their own properties in the offer document.

Every module has it's own functionality and peforms it's own tasks at it's invocation.

• Eligibility: Figuring out if the user is eligible to any of the offers active in the system. Happens every time when a user login event is produced in kafka.

• Redemption: Figuring out if any offer can be redeemed respective to the order placed by the user. Happens everytime when an order is placed by the user.

• Invalidation: Figuring out if any offer has expired due to the order placement. Happens after every time redemption module is run.

Current applications of sc-khata

• Promotions: Standard promotional offers such as Muhurat, New Year offers.
• Zerodha referrals: Referral points from zerodha can be used to get an offer on smallcases.
• Employee offer: Every smallcase employee is eligible for the employee offer under which any smallcase transaction is free of charges.
• case by case: An educational initiative to bring investment lessons and insights from financial experts for free. A small quiz at the end of the session could let the users win prizes such as fees waived off on buying a smallcase.

So sc-khata today is handling enormous number of events, processes numerous offers daily and has made execution of business decisions faster by a significant margin.

In short it has made the investment life a little bit easier!

Saurabh Thakur signing off..!!