1. Turn off the GUI and run the command on the command line

hit CTRL-F1
login
sudo /etc/init.d/gdm stop  // this stops the GUI
stata -b do mydofile.do

2. Run one command after another

stata -b do firstfile.do ; stata -b do secondfile.do ; stata -b do third.do

3. Delay the start of the command

sleep 14400 ; stata -b do myfile.do
#NOTE: sleep takes seconds as argument, so that is 4 hours. Adjust to your needs.

Don’t forget top to check on your processes. Don’t allocate so much memory that you start paging to disk.

Your hints/advice are welcome!

CEC (California Energy Commission) Climate Zones (numbering 16).

zip5 (polygons)

CA counties (polygons)

I used this by using the ArcTool called INTERSECT.

I don’t have a great way of uploading the data, so if you want this data, help me upload it and it will be available to you (and everyone) for free.

http://enoriver.net/index.php/2008/09/24/qualify-your-commands-with-capture/

I’m having to map zip9 (AKA zip+4, zip 9, zip5+4) in California to census block groups (and census tracts) (CBG / CT) and then to latitude and longitude.

I’m having to do this because I haven’t found a similar list online. There are some data vendors, but I’m old-fashioned and like to see the original data sets and do the merge myself.

Odd thing, though, is that I can’t find any zip9 polygons. I just have them as points. Which somewhat makes sense since zip9s are usually a couple of blocks at most, but it’d be nice to know which zip9′s are very large in area and which are small. (Could I back this out by checking the nearest neighbor distance?)

This is a pretty large spatial join since I have a count of 2,854,068 zip9′s in California and something on the order of  10,000-100,000  CBGs.

If you need crosswalks / correspondence tables for this data, please feel free to ask. Preference given to academic researchers and I will ask for a nominal fee (<$50) for time spent preparing it for you, especially if you are not in academia/public-sector.

There is one problem I am encountering, and that is in small samples. If your coefficient of interest has very little variation, be careful. Especially when measuring treatment effects where you have very few or very many treated observations.

Here’s where I had problems: I am conducting a stock market event study. We are trying to figure out what size of event window to use, so we run, for each stock:

return=alpha + beta * market + gamma1 * Day1 +gamma2*Day2 + gamma3*Day3 +epsilon

when we run this with robust, we get much smaller standard errors. We run the Ftest of “test Day1+Day2+Day3=0″ and get significance.

But, when we later run:

return=alpha + beta * market + gamma1to3 * Day1to3 * 1/3 + epsilon,

we don’t get stat significance. And it’s all due to robust. (note, the 1/3 is there to rescale for 3 days)

Delving further, I run a simulation where the dgp is y=x*1/4 + eps

I then regress y on x and a 1day event. I simulate 1000 times and here’s my results:

    Variable |       Obs        Mean    Std. Dev.       Min        Max
-------------+--------------------------------------------------------
          op |      1000   -.0266356    1.000336   -2.86643   3.137967
          os |      1000    1.000005    .0218117   .9318347   1.066719
       opval |      1000    .4983793    .2890897   .0011551    .999441
          rp |      1000   -.0266356    1.000336   -2.86643   3.137967
          rs |      1000    .0435009    .0135115   .0295254   .1079442
-------------+--------------------------------------------------------
       rpval |      1000    .0268978     .121465          0   .9918482

o designates OLS. op is the point estimate for event (should be 0), os is the stderr, and opval is the pvalue.  r designates robust, and rp, rs, and rpval are similarly defined.

Note that robust gives smaller std errors. In fact, much smaller. And this is due to the fact that this coefficient is estimated off one observation.

Importantly, this is NOT fixed if you increase the number of total observations. You have to increase the number of observations in the event. I rerun this with an eventsize of 10, and then OLS and robust are close to matching.

See and run the code to see what I mean.

CODE

* simulation showing how robust raises the t stat for a time series event.
capture program drop eventSim
program define eventSim, rclass
syntax [, obs(integer 1) eventsize(integer 1) xcoeff(real 1) ]
drop _all
set obs `obs'
gen n=_n
gen x=rnormal()
gen y=x*`xcoeff'+rnormal()
gen event=n<=`eventsize'
regress y x
regress y x event
matrix eb=e(b)
matrix ev=e(V)
return scalar OLSpoint=eb[1,2]
return scalar OLSstderr=sqrt(ev[2,2])
test event
return scalar OLSpvalue=r(p)
regress y x event, robust
matrix eb=e(b)
matrix ev=e(V)
return scalar ROBpoint=eb[1,2]
return scalar ROBstderr=sqrt(ev[2,2])
test even
return scalar ROBpvalue=r(p)
end
simulate op=r(OLSpoint) os=r(OLSstderr) opval=r(OLSpvalue) rp=r(ROBpoint) rs=r(ROBstderr) rpval=r(ROBpvalue), reps(1000): eventSim, obs(1000) eventsize(1)  xcoeff(.25)
su
simulate op=r(OLSpoint) os=r(OLSstderr) opval=r(OLSpvalue) rp=r(ROBpoint) rs=r(ROBstderr) rpval=r(ROBpvalue), reps(1000): eventSim, obs(1000) eventsize(10)  xcoeff(.25)
su
** note the mean of os, the mean of rs, and the standard deviation of op should all match.
** the reason is because the dgp is homoskedastic and the robust and ols extimators are both consistent.
** when eventsize is 10, they match pretty well.
** when eventsize is 1, the match is very bad. Robust gives abnormally small standard errors.
** my guess as to why is because the Huber-White robust estimates allow there to be a different sigmasq for the event period.
** When the event is small (or has size 1), then event is perfectly fitted (has zero residual) and that term in the
** heteroskedasticity matrix is small (or zero).

* simulation showing how robust raises the t stat for a time series event.
capture program drop eventSimprogram define eventSim, rclass	syntax [, obs(integer 1) eventsize(integer 1) xcoeff(real 1) ]	drop _all	set obs `obs'	gen n=_n	gen x=rnormal()	gen y=x*`xcoeff'+rnormal()      gen event=n<=`eventsize'	regress y x	regress y x event	matrix eb=e(b)	matrix ev=e(V)	return scalar OLSpoint=eb[1,2]	return scalar OLSstderr=sqrt(ev[2,2])	test event	return scalar OLSpvalue=r(p)
 regress y x event, robust	matrix eb=e(b)	matrix ev=e(V)	return scalar ROBpoint=eb[1,2]	return scalar ROBstderr=sqrt(ev[2,2])	test even	return scalar ROBpvalue=r(p)end
simulate op=r(OLSpoint) os=r(OLSstderr) opval=r(OLSpvalue) rp=r(ROBpoint) rs=r(ROBstderr) rpval=r(ROBpvalue), reps(1000): eventSim, obs(1000) eventsize(1)  xcoeff(.25)susimulate op=r(OLSpoint) os=r(OLSstderr) opval=r(OLSpvalue) rp=r(ROBpoint) rs=r(ROBstderr) rpval=r(ROBpvalue), reps(1000): eventSim, obs(1000) eventsize(10)  xcoeff(.25)su
** note the mean of os, the mean of rs, and the standard deviation of op should all match.** the reason is because the dgp is homoskedastic and the robust and ols extimators are both consistent.** when eventsize is 10, they match pretty well.** when eventsize is 1, the match is very bad. Robust gives abnormally small standard errors.** my guess as to why is because the Huber-White robust estimates allow there to be a different sigmasq for the event period.** When the event is small (or has size 1), then event is perfectly fitted (has zero residual) and that term in the** heteroskedasticity matrix is small (or zero).

Notepad++ is a good alternative. You can still run blocks of code (like control-D) and who do file (like control-R) if you set it up. Plus it’s free.

Here’s my simple 3 step setup.

1) Download and install Notepad++

2) To get context highlighting, go here:  download the link to the stata XML file here http://notepad-plus.sourceforge.net/commun/userDefinedLang/userDefineLang_stata.xml .  Click  start, run, type (or paste in) %APPDATA%\Notepad++ then click ok. Assuming you just installed Notepad++, just copy the file you downloaded to this directory, delete the existing userDefineLang.xml and rename your file userDefineLang.xml.

3) To enable running code from Notepad++, follow steps 1-4 of the following page:

http://s281191135.onlinehome.us/2008/20080427-stata.html

It may look complicated, but it is just downloading two files, saving them to c:\ado\personal (or wherever you like), editing two setup files so it knows which stata version you use, and setting up the F9 ad F10 shortcuts.

That’s it.

You’ll get context highlighting, comment block highlighting, and keyword highlighting.

To change the colors of the highlighting, go to View | User Define Dialog

And follow the directions here: http://notepad-plus.sourceforge.net/uk/UserLanguageDefineSystem-HOWTO.php

Weighted Least Squares and weighted average are opposite concepts, in a sense.

Weighted Least Squares (WLS) is an assumption about your error structure. Let’s say your data is across cities and includes a population variable. If, in STATA, you type:

 regress LHS population otherRHS [aweight=population]

then you are assuming that var(eps_i)=epssq_0*population, AKA variance is higher for large populations, you should give extra weight to the smaller observations. (See Greene, Econometric Analysis)

If instead you want bigger populations to matter more, you’d have to aweight by something inversely proportional, perhaps the inverse of population.

When doing weighted average, you multiply by the population, so bigger cities have more influence. Granted, using a weighted average is restricted to computing averages, it’s not used as part of a regression analysis.

STATA also uses population weighting, which is more like weighted average than not. There, you are assuming precisely var(eps_i)=epssq_0/population. This functional form is driven by the a priori knowledge that your data points represent different sample sizes.

Be wary of anyone who says, “We should have observations with X matter more, so let’s run WLS with X.” They are mixing two concepts.

http://carma.org/

More beta below the bump.

Beta, AKA additional info

Carma is not specifically pitched as a list of power plants in Europe, but it covers Europe well. It is aimed at covering the world. You can download the data.

It has firm names or company names associated with each firm, as well as CO2 emissions, and generation in MWH.

It is not THAT well linked to companies, because some companies have subsidiaries. So, make sure you collect everything with the text “RWE” as an RWE company. I’ve done some linking, so if you want to compare, please shoot me a line and we can share.

You got to be careful of a few things. First, understand that much of Europe uses Combined Heat and Power (AKA cogeneration) and district heating. So, in this case, your tonsCO2/MWH might be high, but this may be due to the CHP. Second, Carma also covers a large amount of auto-generation and might cover refining and heat. For example, International Paper has a plant in Europe. I would guess this is a plant they run to supply their own electricity. Third (and this only matters if you’re charcterizing firms), I believe that many large firms have large partially owned subsidiaries. For example, consider ENBW. It is 45% owned by EDF and 45% owned by a german firm, according to Wikipedia:

The two principal shareholders of EnBW are Électricité de France (EDF), with a 45% share and Oberschwäbischen Elektrizitätswerke (OEW), with a 45% share. In accordance with a closed consortium agreement between EdF and OEW, EdF will take over the supervision over EnBW until 2012. Furthermore, the two shareholders agreed to keep the share ratios as they are and to vote in mutual agreement on important decisions.

Other Data Sources

I’m not sure what their data source is, but I bet that they aggregated info from individual national information. For Europe, Eurostat does NOT have this information (to my knowledge), though Eurostat does report MWH and Carbon on a national basis, I think annually. IEA (International Energy Agency) has the same level of info as Eurostat.

The obvious issue with the Eurostat data is that it aggregates all the power plants in a country. Also a big issue (but not as obvious) is that it is somewhat easy to measure carbon for a country by tracking flows of fuels, but attribution to sectors (electricity, for example) may be problematic. At best, it’s a black box and we have to trust that it was done right.

Carbon from electricity is tracked very well by the CITL (Community Independent Transaction Log) of the European ETS (Emissions Trading Scheme). But this doesn’t report MWH.

Another potential source of info is EPER. But this is emissions only.

In the US, we have the CEMS system, which is continuous emission monitoring system. Data includes heat rate and power output at a high resolution (daily? hourly? I forget). Data is gov’t, so it is public. One problem is that carbon isn’t measured. Not such a big deal, since you can calculate it stoichiometrically from the input fuel information, at least in theory.

I know I should be running things on Unix, but right now I’m on Windows XP. I sometimes putty into a Unix server and run things.

R looks very good. That’s  my next langauge to learn.

Octave is pretty good. It provides syntax almost identical to Matlab.  In 3.0, it now has support for Multidimensional Cell Arrays. These are arrays that can hold any data type. Most common for me is an array of strings. If you load data that is mixed text and numeric, then your data will probably be read as a cell-array.

One thing I have noticed is that the cell-arrays are really quite slow.

I had a ~10000 x 10 csv file.

Column 1 had mixed numeric and strings. They were 6 character codes, and about 2/3 of them did not have alphabetical characters. I needed to convert these to strings, and then do a sort and some other processing. I basically had to traverse each element of the first row and do the datatype change individually.

The process was VERY slow. In fact, I think Excel would be better at such tasks.

Here are a few tips:

• If you can, remove all strings from your CSV file.
• If you read a large dataset as a large cell arrays, separate each column into its own variable. Then pack together the numeric data into a matrix (if needed).
• STATA has an “encode” routine that converts strings into records stored as numeric. For example, if your data range is car makes, it will give each make a number and then also generate a lookup table where you can decipher what the numbers mean.

Also check out this page that benchmarks the math/science packages with a set of standard routines:

http://www.sciviews.org/benchmark/index.html

I couldn’t find such a function in stata. There is no “isnumeric” or “isfloat” function.

There is an extended function called “type”. This is my prefered way to do it:

local mytype : type myvarname

disp "`mytype'"

They also have “confirm”, which works:

http://www.stata.com/help.cgi?confirm

COPIED FROM STATA (accessed 7/28/2009):

You are writing a command that performs some action on each of the
    variables in the local macro varlist.  The action should be different for
    string and numeric variables.  The confirm command can be used here in
    combination with the capture command to switch between the different
    actions:

        foreach v of local varlist {
                capture confirm string variable `v'
                if !_rc {
                        action for string variables
                }
                else {
                        action for numeric variables
                }
        }

    An alternative solution using inline expansion of the extended macro
    function :type (see local) reads

        foreach v of local varlist {
                if substr("`:type `v''",1,3) == "str" {
                        action for string variables
                }
                else {
                        action for numeric variables
                }
        }