Stock imports #1363

2024-10-25T03:13:00+08:00

Shpigford commented

2024-10-25 03:13:00 +08:00

(Migrated from github.com)

Sets the foundation for adding Marketstack as a data provider for non-US stocks.

This is primarily focused on populating the Securities table with all global stocks in the hosted version, but lays the groundwork for pulling prices as well as letting self-hosters only import specific exchanges.

Sets the foundation for adding Marketstack as a data provider for non-US stocks. This is primarily focused on populating the Securities table with all global stocks in the hosted version, but lays the groundwork for pulling prices as well as letting self-hosters only import specific exchanges.

zachgoll (Migrated from github.com) reviewed 2024-10-25 03:49:59 +08:00

zachgoll (Migrated from github.com) left a comment

Setup looks good to me!

app/jobs/securities_import_job.rb

						
				@@ -0,0 +1,13 @@

				class SecuritiesImportJob < ApplicationJob

zachgoll (Migrated from github.com) commented

2024-10-25 03:46:18 +08:00

From a user perspective, my guess is that we're wanting to import by country code correct?

If that's the case, would country_code = nil be a better param to pass? That way, we could do something like this in the job and make it 1 job per user:

def perform(country_code = nil)
  exchanges = StockExchange.in_country(country_code)
  market_stack_client = Provider::Marketstack.new(ENV["MARKETSTACK_API_KEY"])

  exchanges.each do |exchange|
    importer = Security::Importer.new(market_stack_client, exchange.mic)
    importer.import
  end
end

And then just a simple scope on StockExchange:

class StockExchange < ApplicationRecord
  scope :in_country, ->(country_code) { where(country_code: country_code) }
end

From a user perspective, my guess is that we're wanting to import by country code correct? If that's the case, would `country_code = nil` be a better param to pass? That way, we could do something like this in the job and make it 1 job per user: ```rb def perform(country_code = nil) exchanges = StockExchange.in_country(country_code) market_stack_client = Provider::Marketstack.new(ENV["MARKETSTACK_API_KEY"]) exchanges.each do |exchange| importer = Security::Importer.new(market_stack_client, exchange.mic) importer.import end end ``` And then just a simple scope on `StockExchange`: ```rb class StockExchange < ApplicationRecord scope :in_country, ->(country_code) { where(country_code: country_code) } end ```

app/models/provider/marketstack.rb

						
				@@ -0,0 +1,119 @@

				class Provider::Marketstack

zachgoll (Migrated from github.com) commented

2024-10-25 03:34:45 +08:00

Looks like we might be able to consolidate into a fetch_tickers(exchange_mic: nil) method that covers both fetch_all_tickers and fetch_exchange_tickers?

def fetch_tickers(exchange_mic: nil)
  endpoint = "#{base_url}/tickers"
  endpoint = endpoint + "?exchange=#{exchange_mic}" if exchange_mic.present?

  tickers = paginate(endpoint) do |body|
    # ...
  end
end

And then over in importer, would reduce to:

securities = @provider.fetch_tickers(exchange_mic: @exchange)&.tickers

Looks like we might be able to consolidate into a `fetch_tickers(exchange_mic: nil)` method that covers both `fetch_all_tickers` and `fetch_exchange_tickers`? ```rb def fetch_tickers(exchange_mic: nil) endpoint = "#{base_url}/tickers" endpoint = endpoint + "?exchange=#{exchange_mic}" if exchange_mic.present? tickers = paginate(endpoint) do |body| # ... end end ``` And then over in importer, would reduce to: ```rb securities = @provider.fetch_tickers(exchange_mic: @exchange)&.tickers ```

Shpigford (Migrated from github.com) reviewed 2024-10-25 04:19:25 +08:00

app/models/provider/marketstack.rb

						
				@@ -0,0 +1,119 @@

				class Provider::Marketstack

Shpigford (Migrated from github.com) commented

2024-10-25 04:19:25 +08:00

Good catch1

Shpigford (Migrated from github.com) reviewed 2024-10-25 04:26:08 +08:00

app/jobs/securities_import_job.rb

						
				@@ -0,0 +1,13 @@

				class SecuritiesImportJob < ApplicationJob

Shpigford (Migrated from github.com) commented

2024-10-25 04:26:08 +08:00

Good call. Keeps us from having to arbitrarily map mic codes as well.

Shpigford commented

2024-10-25 05:29:24 +08:00

(Migrated from github.com)

@zachgoll Any other changes you'd want made before this gets pushed?

Next PR will be actually setting up appropriate importing based on hosted vs self-hosted. So this on its own doesn't actually make any data additions (just db schema).

@zachgoll Any other changes you'd want made before this gets pushed? Next PR will be actually setting up appropriate importing based on hosted vs self-hosted. So this on its own doesn't actually make any data additions (just db schema).

zachgoll commented

2024-10-25 05:30:19 +08:00

(Migrated from github.com)

@Shpigford yeah I think we're all set for this first step here

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: gavin/maybe#1363

				`@@ -0,0 +1,13 @@`
				`class SecuritiesImportJob < ApplicationJob`