Using Distributed Computing at Scale

  • Product Brief
  • Current Architecture
  • Problem With Data loads and Sleepless Nights
  • Solutions Tried
  • The Final Destination

Product Brief

  • Stock News from multiple sources
  • RSS feed
  • Twitter
  • Reddit
  • Yahoo News
  • Yahoo Finance
  • Earning Calenders
  • Report Filings
  • Company Financials
  • Stock Prices

Current Architecture

Broad View

Problem With Data loads and Sleepless Nights

Solutions Tried

  • Redis sends tasks at high speed but consumers cannot consume at that rate.
  • Redis slows down and lowers the rate at which it sends tasks and now consumers stay idle most of the time.
  • SQS
  • Redis
  • RabbitMQ
  • Zookeeper

The Final Destination

Acknowledgments

--

--

--

Story Teller and Data enthusiast

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Re-balance your Kafka Cluster

“Git” it together: Some tips on commit etiquette and best practices for junior developers

ORIGIN by Cymatics — FREE Retro/Lofi Plugin!

UNIFI DeFi Lending Portal Coming

Cobalt Credits — Unlock Flexible Pentesting

Charging up your command line setup on Windows

How I made my first app

The Adapter Pattern — A Replacement To Dependency Injection

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Japneet Singh Chawla

Japneet Singh Chawla

Story Teller and Data enthusiast

More from Medium

232. Implement Queue using Stacks

Real-Time ETL with Java & Groovy

Connect to Redis in Go

How to Lookup an EAN Barcode Value & Return Product Data Using Go