Bhavik Sachdev

For any given username, the project ingests their most recent GitHub activity including user information, issues, pull requests, commits, stars, and more. The data is not refreshed if the last fetch occurred less than 24 hours ago; this is a trade-off between reducing API usage and keeping the data reasonably up to date. Since the data store is primarily used for analytics or awarding users based on their activity, real-time updates are not necessary. The data is only refreshed if the same user is re-ingested or if a query requests an update and the existing data is more than 24 hours old. The project includes measures such as rate limiting and a retry mechanism for reducing the usage of the Github API. Both of these are implemented by overriding a couple of methods in the standard Go http.Client. The requests sent to the engine are queued and a worker pool picks them up. The limit of the queue is 1000 and we have 10 workers (defaults), this allows the engine to ingest a lot of data at once. image