Data Sources

← Back to shibui.finance

Prices, statements, and valuations

A licensed financial data provider supplies daily prices, financial statements, and valuation metrics. Data is loaded into the database daily via API.

Dataset	History	Notes
Daily prices (OHLCV)	1962 - present	~31M rows. Split-adjusted. All NYSE + NASDAQ listed securities.
Financial statements	1990 - present	Quarterly and annual. Income statement, balance sheet, cash flow. 70+ line items per filing.
Valuation metrics	1993 - present	Daily: market cap, enterprise value, P/E, P/B, trailing PEG.
Derived daily ratios	1993 - present	Computed from prices and statements: EV/EBIT, EV/EBITDA, P/S, P/CF, FCF yield, dividend yield, payout ratio, earnings yield.
Derived quarterly ratios	1990 - present	Computed from statements: asset turnover, EBITDA margin, ROIC, working capital, ROE TTM, profit margin TTM.
Technical indicators	1962 - present	56 indicators computed from prices. Includes RSI, MACD, Bollinger Bands, SMA, EMA, ADX, ATR, Stochastic, CCI, OBV, 15 candlestick patterns, and more.
Company metadata	Current	Sector/industry (Morningstar classification), SIC codes, company website, reporting currency.

Analyst data, classification, and ownership

A second licensed data provider supplements fields not covered by the primary source, including GICS classification, analyst estimates, earnings history, and ownership statistics.

Dataset	Coverage	Notes
GICS classification	Current	Sector, industry group, industry, sub-industry. Primary source for GICS taxonomy.
Interest income	Sparse pre-2019, ~60% from 2019+	Interest income and net interest income. Not available from the primary source.
Analyst estimates	Current snapshot	Wall Street target prices, consensus EPS estimates (current and next year/quarter).
Earnings history	Quarterly	EPS actuals, estimates, surprise %, before/after market timing.
Ownership statistics	Current snapshot	Insider %, institutional %, short interest %, shares float.
Company details	Current	Business descriptions, HQ location, employee count, IPO date, CIK, ISIN.

Currency normalization

Some companies listed on US exchanges report financial statements in their home currency. When supplementary data is sourced in a foreign currency, the pipeline converts it to USD using daily exchange rates from the Federal Reserve Economic Data (FRED) API.

The FRED rates cover 18 currencies: AUD, BRL, CAD, CHF, CNY, DKK, EUR, GBP, HKD, INR, JPY, KRW, MXN, MYR, SEK, SGD, TWD, and ZAR. Rates go back to 2000, updated daily.

For each affected statement, the conversion uses the closest available FRED rate on or before the filing date. Approximately 700 securities are affected - mostly foreign companies (ADRs and foreign private issuers) whose supplementary data arrives in the local reporting currency. If no matching exchange rate is available, the field is set to NULL rather than left in the wrong currency.

All prices, valuations, and primary financial statement data are already denominated in USD from the primary data provider. The FRED conversion applies only to supplementary fields (interest income, net interest income) sourced from a secondary provider.

SEC EDGAR filings

Filing metadata from the SEC EDGAR database. This is public government data, not a licensed provider. The table contains filing dates, form types, accession numbers, and direct URLs to the SEC filing page. It does not contain the filing text itself (no 10-K narrative, no Form 4 transaction details).

6.4M rows, 1993 to present. Updated near-real-time from the EDGAR feed (typically within minutes of a filing appearing on SEC.gov).

Filing category	Volume	Notes
Annual reports (10-K)	75K filings	Annual financial reports. Use report_date for the fiscal period end.
Quarterly reports (10-Q)	211K filings	Quarterly financial reports.
Current reports (8-K)	762K filings	Material events. Includes item codes identifying the event type (e.g., officer changes, earnings results).
Insider transactions (Form 4)	2.6M filings	Insider buy/sell filings. 41% of all rows. Metadata only: dates and URLs, not transaction details.
Beneficial ownership (SC 13G/13D)	405K filings	Ownership above 5% threshold. Includes both passive (13G) and activist (13D) filings.
Proxy statements (DEF 14A)	74K filings	Definitive proxy statements for shareholder meetings.
Foreign issuer reports (6-K, 20-F)	317K filings	Current and annual reports from foreign private issuers listed on US exchanges.
Prospectus supplements (424B2)	645K filings	10% of all rows. Dominated by financial sector shelf offerings. Filter by form_type to exclude.

See how analysts triage the filing feed by size and event type with Claude.

Classification and listings

• Nasdaq (FTP + Screener API): Ticker symbols, listing exchange, country of origin. Nasdaq sector/industry data serves as a third-tier fallback for GICS classification when other sources are unavailable.
• OpenFIGI: Security type classification (Common Stock, ETF, ADR, REIT, Preferred Stock, Closed-End Fund, MLP). Used to distinguish equity types and filter non-equity instruments.

How GICS sectors are assigned

GICS (Global Industry Classification Standard) sector and industry codes are assigned through a three-tier fallback process:

1. Direct GICS (primary): GICS sector, industry group, industry, and sub-industry from the supplementary data provider.
2. Morningstar-to-GICS mapping (fallback): The primary data provider supplies Morningstar sector and industry classifications. A 166-entry mapping table converts these to the GICS hierarchy. Used when direct GICS data is missing.
3. Nasdaq-to-GICS mapping (last resort): Nasdaq screener sector mapped to GICS sector only (no industry or sub-industry resolution). 13-entry mapping. Used when both direct GICS and Morningstar data are unavailable.

Approximately 4,150 securities (mostly ETFs, closed-end funds, and preferred stocks) lack GICS classification entirely.

Optional filters

AAOIFI Shariah compliance filter: A static exclusion list based on MSCI GICS-level screening rules. Excludes sectors and sub-industries incompatible with Islamic finance principles (conventional banking, alcohol, gambling, tobacco, weapons, entertainment). 13 exclusion rules mapped to GICS sector or sub-industry codes. Available as a reference table for filtering; not applied by default.

Coverage

• 9,900+ securities across NYSE and NASDAQ
• ~4,800 common stocks, ~3,800 ETFs, 375 ADRs, 364 closed-end funds, 187 REITs
• US equities only
• End-of-day data with approximately one trading day lag
• Daily automated refresh, typically complete by 6-8 AM UTC
• SEC EDGAR filing metadata: 6.4M filings across 342 form types, 1993 to present, updated near-real-time

Limitations

Contact

Questions about data coverage or accuracy? chris@shibui.finance