data/input/ stores all datasets fed into the do files as inputs. 
data/output/ stores all intermediate and final versions of the datasets.

R: Run build_index.R then data_extract.R. 
R/ stores the R code used to scrape the intial IRS990 data. 
R/build_index.R takes a list of EINs for the organizations we are interested in and feeds them to Amazon Web Servers (AWS) to get a list of directories to search for each organization-year's IRS990 form
R/build_core_modified.R defines a function called build_core that scrapes the data when fed an appropriate index dataframe.
R/data_extract.R feeds the index data frame built by build_index.R to the function build_core_modified.R. It saves the scraped data as a csv. 
R/archive/build_core_demo.R was a smaller version of build_core.R made for testing purposes. 

python: Run wikiscrape.py
python/wikiscrape.py scrapes the first paragraph of each think tank's page

STATA: Run IRS990Master.do. 
STATA/IRS990Master.do runs all cleaning and analysis files in order
STATA/prep/IRS990cleaning.do performs basic prep and generates a few fundamental variables
STATA/prep/IRS990namesdata.do merges in the data we have regarding the gender of each employee, and it generates the key gender-related variables at the individual and organizational level
STATA/prep/IRS990wikidata.do merges in the data we have regarding political affiliation (scraped from wikipedia) and year of founding
STATA/analysis/IRS990_organization_level.do generates a report about statistics mostly at the organization level.
STATA/analysis/IRS990_individual_level.do generate graphics and analysis mostly at the individual level. 
STATA/analysis/IRS990_factoids.do generates a few miscellanous facts used in the note / blog

resources/ contains (1) PDFs with rankings data that we gathered manually, (2) information on the IRS990 form, and (3) information on XML schemas used in the scraping process. 

writing/ contains our final outputs
writing/Figures/ contains individual figures. The main ones used in the report are "cibar_politics", "payratio2016_founding", "2016paydist_gender", "rep_byposition", and "2016pos_paydist_gender"

contact jduggan@cgdev.org 