Skip to content

andre-christi-kan, portfolio

Kadence International, full-time

Kadence Kepo

  • Android
  • Web

Kadence Kepo is Kadence's field-operation app for its registered shoppers. The app sends GPS when a visit starts and again when the shopper marks the visit visited. Photos, videos and audio stay on the phone until upload, then go to the API in 50 MB chunks and on to AWS S3. When the app's traffic is full, shoppers upload on a separate web upload page. Kepo has 1k–2k users and ran 3 fieldwork projects at once on one small server.

Where
Kadence International, full-time employment
Role
Lead Engineer. I built every service and the Android app, and ran the infrastructure.
Team
Me, 1 teammate, who built the front end of the web dashboard and the web upload page, and 1 designer, who designed the UI. I reviewed the team's code and coordinated with project managers, QA and operations.
Dates
Jul 2021 – Dec 2021, about 6 months at the start of my time at Kadence
Platforms
Android (outside Google Play and the App Store), web dashboard for operations, web upload page for shoppers
Stack
NestJS microservices, RabbitMQ, MongoDB, AWS S3, Flutter
upload chunks, each sent to AWS S3 as one part
50 MB
one server, with 50 GB of storage
2 GB RAM / 1 vCPU
fieldwork projects at once on that server
3

01Context and my role

I led Kepo as Lead Engineer, in about 6 months at the start of my time at Kadence. I built the user service and the Kepo service, the Android app and the API behind the web upload page, and ran the infrastructure. My teammate built the front end of the web dashboard, where operations review the files, and of the web upload page. A designer designed the UI.

Kadence Kepo is a separate project from Field Operation Platform, my freelance field-operation app, and is not Kepo Solusi Indonesia, where I had my first backend role.

  1. Jul 2021Project starts

    At the start of my time at Kadence. I built the user service first.

  2. Dec 2021Work ends

02What it does

  • Visits with GPS

    When a shopper starts a visit, the app sends a GPS check-in to the API right away, and again when the shopper marks the visit visited.

  • Capture first, upload after

    Photos, videos and audio stay on the phone until upload.

  • Uploads in 50 MB chunks

    Each file is a chunked upload. The app sends it to the API in 50 MB chunks. The API passes each chunk on to AWS S3 as one part, then completes the file on S3.

  • Retry per file

    Each file has a retry button. A failed upload is sent again from the start.

  • Web upload page

    When the app's traffic is full, shoppers upload in a browser on a separate web upload page.

  • Operations review

    Operations review the files in the web dashboard. My teammate built its front end.

  • For registered shoppers

    Only Kadence's registered shoppers use Kepo, so it has 1k–2k users. The Android app is distributed outside Google Play and the App Store.

03How it works

Kadence Kepo and the systems around itThe Kepo service and the user service are NestJS microservices that talk to each other through RabbitMQ. Shoppers use the Flutter Android app for visits, photos, videos and audio, and the app sends GPS and 50 MB chunks to the Kepo service. The Kepo service sends each chunk on to AWS S3 as one part and reads and writes MongoDB, where each service has its own database. Operations review the files in the web dashboard. When the app's traffic is full, shoppers upload in a browser on the web upload page. The page sends the file to its own API, outside the microservices, which updates the task data on the Kepo service over HTTP.visits, photo,video, audiowhen app trafficis fullGPS, 50 MB chunksreview filesthe fileupdates task data,over HTTPservicemessagesservicemessageseach chunkas a partreads, writesShoppersAndroid appFlutterWeb upload pageWeb dashboardfor operationsUpload page APIMongoDBone database per serviceAWS S3Kepo serviceUser serviceRabbitMQ
The Kepo service and the user service are NestJS microservices that talk to each other through RabbitMQ, and each service has its own MongoDB database. The Android app sends GPS and 50 MB chunks to the Kepo service, which sends each chunk on to AWS S3 as one part. Operations review the files in the web dashboard. When the app's traffic is full, shoppers upload on the web upload page, whose own API updates the task data on the Kepo service over HTTP.
  • A visit: the shopper starts a visit, and the app sends GPS to the Kepo service right away. Photos, videos and audio are saved on the phone. When the shopper marks the visit visited, the app sends GPS again.
  • Upload: the app sends each file to the Kepo service in 50 MB chunks. The service forwards each chunk to AWS S3 as one part (S3 multipart, part by part through the API) and completes the file on S3 after the last part.
  • Failed upload: the shopper taps retry on that file, and the app sends the file again from the first chunk.
  • Web upload page: when the app's traffic is full, shoppers open the web upload page in a browser. The page sends the file to its own API, which I built. That API then updates the task data on the Kepo service over HTTP. My teammate built the page.
  • Services: Kepo ran as its own service, the Kepo service, in Kadence's NestJS microservices backend, next to the user service for sign-up and sign-in. I built the user service first, during Kepo. The services talk to each other over RabbitMQ, the message broker between them. Each service has its own MongoDB database. Kadence K-OINS later ran on the same backend.
  • Server: one server with 2 GB RAM, 1 vCPU and 50 GB of storage ran 3 Kadence fieldwork projects at the same time, with shoppers uploading and recording at once. With memory optimization, the app ran on that server.
  • Operations: operations review the files in the web dashboard. My teammate built its front end.

Visit and upload

A Kadence Kepo visit and its uploadThe shopper starts a visit in the Android app, which sends GPS to the Kepo service right away. Photos, videos and audio are saved on the phone until upload. When the shopper marks the visit visited, the app sends GPS again. For each 50 MB chunk, the app sends the chunk to the Kepo service, which sends it on to AWS S3 as one part. The Kepo service then completes the file on S3 and tells the app the file is stored. If an upload fails, the app tells the shopper, the shopper taps retry on that file, and the app sends the file again from the first chunk.AWS S3Kepo serviceAndroid appShoppersaved onthe phoneuntil uploadrepeats foreach chunkopt[if it fails]start visitGPS, visit startphoto, video,audiomark visitedGPS, visiteda 50 MB chunkthe chunk,as one partcomplete the filefile storedupload failedtaps retryon the filethe file again,from the first chunk
The app sends GPS when a visit starts and again when it is marked visited, and saves photos, videos and audio on the phone until upload. For each 50 MB chunk, the app sends the chunk to the Kepo service, which sends it on to AWS S3 as one part, then completes the file on S3. If an upload fails, the shopper taps retry on that file and the app sends the file again from the first chunk.

04Decisions and trade-offs

Uploads

Uploads through the API

Why
At the time I did not know about presigned URLs, and I wanted every upload to go through one place. The app sends each file to the Kepo service in 50 MB chunks, and the service sends each chunk on to AWS S3 as one part.
Cost
In hindsight it was the wrong call. Every chunk passes through the API server.

A separate upload path for traffic peaks

Why
When the app's traffic is full, shoppers can still upload on the web upload page. Its own API receives the file and updates the task data on the Kepo service over HTTP.

Infrastructure

One small server

Why
To keep the cost low. 3 fieldwork projects at once ran on 2 GB RAM, 1 vCPU and 50 GB of storage, with memory optimization.

Its own service in a microservices backend

Why
The user service holds sign-up and sign-in, so I built it first, and the Kepo service sits next to it. The services talk over RabbitMQ, each with its own MongoDB database. K-OINS later ran on the same backend.

05Stack

Backend
  • NestJS microservices: the Kepo service and the user service
Messaging
  • RabbitMQ, between the services
Data
  • MongoDB, one database per service
Files
  • AWS S3 (S3 multipart, part by part through the API)
Android app
  • Flutter