Source: MCXC J0106.8+0103
Observation ID: 0762870601
SAS Version: 19.0.0
Python version: 3
csh is used
Everything you need in this tutorial is in:
cheese.zip
.
You work in the main folder cheese, and put the ccf.cif, SAS files and products from previous chapter like mos1S001-clean.fits in this folder. There are several subfolders:
1. src - the source code is found there. src/cheese_script.py is what you need in this tutorial.
2. log - an empty folder for you to put the log files.
3. command - the subcommands for the task cheese. Every time you run an ESAS command, there is a temporary file called command.csh which contains all the subcommands. You can run the subcommands one by one.
4. esas_products - this is the folder I work on. The content is the same as the main folder but with all the files needed for and created by this chapter. Please note that some files have several updates. Those in the folder are files from the final update.
In this tutorial, I only use mos2S002 as an example.
Masking Point Sources
In ESAS, cheese is used to mask point sources. Another common way to do is ewavelet. Now let's set the ccf and run cheese:
setenv SAS_CCF ccf.cif
cheese prefixm="1S001 2S002 " prefixp=S003 scale=0.25 rate=0.0001 dist=0 clobber=1 elow=400 ehigh=2300 > log/cheese.log
and you find the final products you need – mos2S002-bkg_region-det.fits, mos2S002-bkg_region-sky.fits and mos2S002-cheese.fits. However, for SAS 19.0.0, cheese automatically deletes and renames some files upon completion. If you want to understand better what happens. I suggest running the subcommands in command/cheese.csh one by one. In this tutorial, I use the products produced from this script. I will first show the individual commands and then I will show you how to do the whole thing fast using the python script src/cheese_script.py.
1.Step by step guide
Here are the basic steps in cheese.
Open the python script and see what you need.
Here are the basic steps in cheese.
1.Produce the following for point source detection:
a) an image extracted from mos2S002-clean.fits in a certain energy range of your choice
b) an exposure map in the same energy range
c) create a “mask” image from the exposure map on which the point sources search will perform. This is an image indicating the valid region for the search. CCDs and bad pixels are invalid regions with a value = 0, and all other parts = 1.
2. run eboxdetect for the first time for local detection using the products produced in the previous step to produce a file called eboxlist_l.fits. This file contains detected point sources information.
3. Run esplinemap using eboxlist_l.fits to produce a background map called mos2S002-obj-imbkg.fits from non-source regions.
4. Run eboxdetect for the second time for map detection using the background map produced from the previous step. A file called eboxlist_m.fits is produced, which has some overlap with eboxlist_l.fits.
5. Run emldetect to perform psf fitting on the point sources in eboxlist_m.fits. The final sources satisfying selection criteria is put in emllist.fits.
6. Run region twice to create two fits files which contain the size and coordinates of the detected point sources satisfying selection criteria in sky and detector coordinates. The files are called mos2S002-bkg_region-sky.fits and mos2S002-bkg_region-det.fits. mos2S002-bkg_region-det.fits is used to mask point sources when producing a spectrum. mos2S002-bkg_region-sky.fits is used to create a mask in the image form which is used when handling image data in the next step. The point sources detected are different for different ccds. If you want the same mask for all ccds, update mos2S002-bkg_region-sky.fits and mos2S002-bkg_region-det.fits in this step.
7. Run make_mask to translate the coordinates and sizes of the point sources in mos2S002-bkg_region-sky.fits to an image. The final product is mos2S002-cheese.fits. This file is used when handling image data.
8. Remove or add a mask. The source you feel interested in is probably masked. You have to remove it. Or if you want to enlarge the size of a certain mask or even add an extra mask, do it in this step.
Here comes the details:
1.Production of an image, exposure map and mask in a certain energy range.
For the energy range, choose the one you work on. My example is [0.3 – 2.4] keV, the range I use for image data. Of course, we use a far wider range for spectroscopic data, but using the narrow range would result in no difference, I believe.
Here are the steps for producing the image, exposure map and mask. All are in src/cheese.csh.
Image
evselect table=mos2S002-clean.fits:EVENTS withfilteredset=yes expression='(PATTERN<=12)&&(FLAG == 0)&&(PI in [400:2300])' filtertype=expression imagebinning='imageSize' imagedatatype='Int32' imageset=mos2S002-obj-im.fits squarepixels=yes ignorelegallimits=yes withxranges=yes withyranges=yes xcolumn='X' ximagesize=900 ximagemax=48400 ximagemin=3401 ycolumn='Y' yimagesize=900 yimagemax=48400 yimagemin=3401 updateexposure=yes filterexposure=yes verbosity=1
Fig.1 an image extracted from mos2S002-clean.fits in [0.4 – 2.3] keV. This image shows the photons in this energy range.
exposure map
eexpmap imageset=mos2S002-obj-im.fits attitudeset=atthk.fits eventset=mos2S002-clean.fits expimageset=mos2S002-obj-imexp.fits withdetcoords=no withvignetting=yes usefastpixelization=no usedlimap=no attrebin=4 pimin=400 pimax=2300
Fig.2 an exposure map in [0.4 – 2.3] keV. This has been corrected for vignetting effect. Otherwise, all pixels have the same value.
mask
emask expimageset=mos2S002-obj-imexp.fits detmaskset=mos2S002-obj-immask.fits detmasktable=MASK threshold1=0.3 threshold2=0.5
Fig.3 A mask indicating ccd gaps and bad pixels. All pixels have either a value of 0 or 1. Point source searches only perform on pixels with a value = 1.
2. run eboxdetect for the first time for local detection to produce a file called eboxlist_l.fits. This file contains detected point sources information.
eboxdetect detmasksets='pnS003-obj-immask.fits mos1S001-obj-immask.fits mos2S002-obj-immask.fits' withdetmask=yes expimagesets='pnS003-obj-imexp.fits mos1S001-obj-imexp.fits mos2S002-obj-imexp.fits' withexpimage=yes nruns=3
likemin=15
boxsize=5 ecf='3.2 1.2 1.2' imagesets='pnS003-obj-im.fits mos1S001-obj-im.fits mos2S002-obj-im.fits' boxlistset=eboxlist_l.fits hrdef='1 2 2 3 3 4' pimin='400 400 400' pimax='2300 2300 2300' obsmode=pointing
The critical parameter here is likemin, which determines the minimum detection likelihood. Now you can take a look at this file:
fv eboxlist_l.fits &
It has 33 columns and 268 rows. For the column ID_INST,0=summary,1=pn,2=mos1,3=mos2. Check other parameters yourself.
3. Run esplinemap using eboxlist_l.fits to produce a background map called mos2S002-obj-imbkg.fits from non-source regions
esplinemap
scut=0.01 mlmin=1
nsplinenodes=20 excesssigma=4 nfitrun=3 idband=1 boxlistset=eboxlist_l.fits imageset=mos2S002-obj-im.fits expimageset=mos2S002-obj-imexp.fits withexpimage=yes detmaskset=mos2S002-obj-immask.fits withdetmask=yes bkgimageset=mos2S002-obj-imbkg.fits pimin=400 pimax=2300 fitmethod=spline snrmin=30 smoothsigma=15
The important parameters here are scut and mlmin. scut is the source cut level in counts/arcsec**2. mlmin is the minimum detection likelihood. The product of this command is mos2S002-obj-imbkg.fits.
Fig.4 mos2S002-obj-imbkg.fits, A background map produced from esplinemap.
4. Run eboxdetect for the second time for map detection using the background map produced from the previous step(mos2S002-obj-imbkg.fits). A file called eboxlist_m.fits is produced, which has more sources (336) than eboxlist_l.fitseboxdetect bkgimagesets='pnS003-obj-imbkg.fits mos1S001-obj-imbkg.fits mos2S002-obj-imbkg.fits' usemap=yes usematchedfilter=no detmasksets='pnS003-obj-immask.fits mos1S001-obj-immask.fits mos2S002-obj-immask.fits' withdetmask=yes expimagesets='pnS003-obj-imexp.fits mos1S001-obj-imexp.fits mos2S002-obj-imexp.fits' withexpimage=yes nruns=3
likemin=15
boxsize=5 withimagebuffersize=no imagebuffersize=640 ecf='3.2 1.2 1.2' imagesets='pnS003-obj-im.fits mos1S001-obj-im.fits mos2S002-obj-im.fits' boxlistset=eboxlist_m.fits withoffsets=no mergedlistset=mergedlist.fits hrdef='1 2 2 3 3 4' pimin='400 400 400' pimax='2300 2300 2300' obsmode=pointing
Again the important parameter is likemin. Lowering this would result in more sources detected.
5. Run emldetect to perform psf fitting on the sources in eboxlist_m.fits. The final sources satisfying selection criteria is put in emllist.fits.
emldetect boxlistset=eboxlist_m.fits
mllistset=emllist.fits mlmin=10
determineerrors=yes fitposition=yes psfmodel=ellbeta nmaxfit=1 nmulsou=1
ecut=15 scut=15
fitextent=yes extentmodel=beta dmlextmin=6 minextent=1.5 maxextent=20 withthreshold=yes threshold=15 threshcolumn=LIKE withtwostage=yes withimagebuffersize=no imagebuffersize=640 withxidband=no xidfixed=no rateonly=no simulate=no pimin='400 400 400' pimax='2300 2300 2300' hrpndef='1 2 2 3 3 4 4 5' xidpndef='2 3 4' hrm1def='1 2 2 3 3 4 4 5' xidm1def='2 3 4' hrm2def='1 2 2 3 3 4 4 5' xidm2def='2 3 4' ecf='3.2 1.2 1.2' xidecf=1 fitcounts=yes fitnegative=no mergedlistset=mergedlist.fits useevents=no usecalpsf=yes withhotpixelfilter=no withoffsets=no withrawrows=no tmpwrite=0 imagesets='pnS003-obj-im.fits mos1S001-obj-im.fits mos2S002-obj-im.fits' bkgimagesets='pnS003-obj-imbkg.fits mos1S001-obj-imbkg.fits mos2S002-obj-imbkg.fits' detmasksets='pnS003-obj-immask.fits mos1S001-obj-immask.fits mos2S002-obj-immask.fits' withdetmask=yes expimagesets='pnS003-obj-imexp.fits mos1S001-obj-imexp.fits mos2S002-obj-imexp.fits' withexpimage=yes
sourceimagesets='pnS003-obj-imsmap.fits mos1S001-obj-imsmap.fits mos2S002-obj-imsmap.fits'
withsourcemap=yes
The most important parameter is again mlmin (minimum detection likelihood). scut (source selection radius for fitting) and ecut (source cut-out radius) may also have some effect in the result of source detection.
Now all detected sources are in emllist.fits. Let’s open it:
fv emllist.fits &
It has got 46 columns, including those in eboxlist_m.fits An important new column here is
DET_ML, which is the detection likelihood. It will be used in the final extraction of the mask. When you add up this value of different ccds under the same ID_CLUSTER, it is roughly the summary column (i.e. ID_INST = 0).
To make it easier to read, let’s output the important columns which will be used in the final mask extraction into a file called emllist.txt:
fdump emllist.fits+1 emllist.txt "ID_INST ID_BAND ID_CLUSTER FLUX DET_ML DIST_NN" - pagewidth=256 prhead=no clobber=yes
(src/cheese_script.py: a.fitstotxt() for the above command)
Now let’s open the source images which display all the sources detected in emllist.fits. These images are products of emldetect.They are the same for all ccds.
ds9 pnS003-obj-imsmap.fits mos1S001-obj-imsmap.fits mos2S002-obj-imsmap.fits - frame lock image -zoom to fit -zscale -frame prev -zscale -frame prev -zscale &
Fig.5 pnS003-obj-imsmap.fits,mos1S001-obj-imsmap.fits and mos2S002-obj-imsmap.fits. The images display the sources detected in emllist.fits.
Now you can test by running emldetect again, but change mlmin to a larger value, e.g. 150, and open mos2S002-obj-imsmap.fits. You find there are a lot less sources.
6&7. Now we have finalized the sources in emllist.fits. We have to make further cuts to have the final list. Run region twice to create two fits files which contain the size and coordinates of the detected point sources satisfying selection criteria in sky and detector coordinates. The files are called mos2S002-bkg_region-sky.fits (for image data, this file needs further processing) and mos2S002-bkg_region-det.fits (for spectroscopic data).
Remember at the beginning you input some parameters when calling cheese? Here are when these parameters come into play!
region eventset=mos2S002-clean.fits operationstyle=global srclisttab=emllist.fits:SRCLIST expression='
(FLUX >= 1e-18)
&&
(DET_ML >= 15)
&&
(ID_INST == 3)
&&
(DIST_NN >= 0)
&&(ID_BAND == 1)' bkgregionset=mos2S002-bkg_region-det.fits
bkgfraction=0.25
radiusstyle=contour nosrcellipse=no
outunit=detxy
verbosity=1
region eventset=mos2S002-clean.fits operationstyle=global srclisttab=emllist.fits:SRCLIST expression='(DIST_NN >= 0)&&(DET_ML >= 15)&&(ID_INST == 3)&&(FLUX >= 1e-18)&&(ID_BAND == 1)' bkgregionset=mos2S002-bkg_region-sky.fits radiusstyle=contour nosrcellipse=no bkgfraction=0.25
outunit=xy
verbosity=1
When you run cheese, you input: scale=0.25 rate=0.0001 dist=0. scale is the bkgfraction, which means the source is removed down to a level where the surface brightness of the source is one quarter of the surrounding background. rate=0.0001 corresponds to FLUX >= 1e-18. rate=1 is 10**14 ergs /cm**2/s. and dist=0 is DIST_NN >= 0, which is the minimum separation for sources in arc seconds (The cookbook suggests a default of 50, and can be as large as 1000). outunit is the source coordinate. detxy for detector and xy for sky coordinates.
For DET_ML >= 15, it is the detection likelihood limit. ID_INST indicates the instrument, with pn= 1, mos1=2, mos2=3. In this case, different ccds have their own mask since for the same source, they have a different DET_ML. Now let’s check how well the source mask masks sources. To begin with, let’s produce the rate image by dividing the counts image, mos2S002-obj-im.fits, by the exposure, mos2S002-obj-imexp.fits:
farith mos2S002-obj-im.fits mos2S002-obj-imexp.fits mos2S002-rate.fits DIV
(bin/cheese_script.py : a.makeImg() for the above)
Note that we don’t correct for oot events for pn here.
Next we have to convert mos2S002-bkg_region-sky.fits into ds9 readable format. The output is mos2_sky.reg. In src/cheese_script.py, you do it with this line:
a.regionextract("mos2S002-bkg_region-sky.fits",”mos2_sky.reg”)
and the output is a text file with the coordinates and radii called mos2_sky.reg. If you convert other ccds and also the files for detector coordinates (mos2S002-bkg_region-det.fits), you find for sky coordinates, different ccds have the same coordinates but not the detector ones. The radii are different for different ccds due to psf fitting.
Let’s take a look to learn better.
emacs mos1_sky.reg mos2_sky.reg pn_sky.reg
emacs mos1_det.reg mos2_det.reg pn_det.reg &
Then we open rate-mos2S002.fits with the point sources of mos2S002-bkg_region-sky.fits loaded. You can also compare with mos2S002-obj-imsmap.fits, which shows the point sources not passing the selection criteria.
ds9 mos2S002-rate.fits -scale limits 0 5e-5 -zoom to fit -smooth -region load mos2_sky.reg mos2S002-obj-imsmap.fits -zoom to fit -zscale &
Fig.6 left: rate-mos2S002.fits. right: mos2S002-obj-imsmap.fits, an image showing all sources in emllist.fits.
If you think it is not good, then do region again, lowering the threshold of DET_ML.
Let’s check further all the ccds.
ds9 rate-mos1S001.fits -scale limits 0 5e-5 -smooth -region load mos1_sky.reg rate-mos2S002.fits -scale limits 0 5e-5 -smooth -region load mos2_sky.reg rate-pnS003.fits -scale limits 0 5e-5 -zoom to fit -smooth -region load pn_sky.reg -frame prev -zoom to fit -frame prev -zoom to fit &
Fig.7 top left: rate-mos1S001.fits,top right: rate-mos2S002.fits, bottom left: rate-pnS003.fitspn finds a lot more point sources! I believe the reason is pn detects photons a few times higher than mos1 and mos2, resulting in a higher detection likelihood. Now let’s load the rate images of mos2 but with the pn region file loaded, and compare with the one you just loaded:
ds9 rate-mos2S002.fits -scale limits 0 5e-5 -smooth -region load mos2_sky.reg rate-mos2S002.fits -scale limits 0 5e-5 -smooth -region load pn_sky.reg -zoom to fit -frame prev -zoom to fit &
Fig.8 Close-up of rate-mos2S002.fits. The left one is loaded with mos2’s mask. The right one is loaded with pn’s mask. Green ellipses are common masks to both, cyan only found in mos2 and red only found in pn.
In the above figure, you see in some red ellipses, there is a faint source, but just too dim to pass the selection criteria. The opposite also happens. There are sources found in mos2 but not pn. If you want all ccds having the same mask, we have to do region again but choose the summary band, i.e. (ID_INST ==0 )&&(ID_BAND == 0). But if you open emllist.fits, you find for ID_BAND=0, there are a lot of FLUX with “INDEF”.
Let’s take a look:
fdump emllist.fits+1 oldflux.txt "ID_INST FLUX" - pagewidth=256 prhead=no clobber=yes
If that mask is detected in one ccd (e.g. mos2 or pn), but falls on ccd gaps or missing ccds in another(e.g. mos1), the summary band would be “INDEF”. So, in order to use this band, we have to update emllist.fits first so that there is no INDEF value for the summary band. In src/cheese_script.py, you find the following line:
a.emllistUpdate()
The FLUX of the summary band is roughly the average of the ccds.
Now check emllist.fits again:
fdump emllist.fits+1 newflux.txt "ID_INST FLUX" - pagewidth=256 prhead=no clobber=yes
Compare newflux.txt and oldflux.txt, and you see FLUX is successfully updated.
Now we do region again to extract mos2S002-bkg_region-det.fits and mos2S002-bkg_region-sky.fits, this time using the summary band,(ID_INST == 0)&&(ID_BAND == 0):
region eventset=mos2S002-clean.fits operationstyle=global srclisttab=emllist.fits:SRCLIST expression='(FLUX >= 1e-18)&&(DET_ML >= 15)&&(ID_INST == 0)&&(DIST_NN >= 0)&&(ID_BAND == 0)' bkgregionset=mos2S002-bkg_region-det.fits bkgfraction=0.25 radiusstyle=contour nosrcellipse=no outunit=detxy verbosity=1
region eventset=mos2S002-clean.fits operationstyle=global srclisttab=emllist.fits:SRCLIST expression='(DIST_NN >= 0)&&(DET_ML >= 15)&&(ID_INST == 0)&&(FLUX >= 1e-18)&&(ID_BAND == 0)' bkgregionset=mos2S002-bkg_region-sky.fits radiusstyle=contour nosrcellipse=no bkgfraction=0.25 outunit=xy verbosity=1
then make the cheese image again:
make_mask inimage=mos2S002-obj-im.fits inmask=mos2S002-mask-im.fits outmask=mos2S002-cheese.fits reglist=mos2S002-bkg_region-sky.fits (src/cheese_script.py:a.remakeRegion())
Now you open mos1S001-cheese.fits, mos2S002-cheese.fits and pnS003-cheese.fits, they all have the same masks. Sources detected in one ccd are also masked in other ccds.
ds9 pnS003-cheese.fits mos1S001-cheese.fits mos2S002-cheese.fits -frame lock image -zoom to fit &
Fig.9 pnS003-cheese.fitsmos1S001-cheese.fits and mos2S002-cheese.fits. After update, all ccds have the same masks.
8. remove or add a mask
a. How to remove a mask
From the above, we see the cluster in the centre is also masked. This is not what we want, we now have to remove it. The easiest way is to open mos2S002-bkg_region-sky.fits and mos2S002-bkg_region-sky.fits with fv, and then manually change the radii to 0.
If you don’t want to do it manually, let’s do it with other commands in ftools. First, we need to decide which mask we have to remove, we convert mos2S002-bkg_region-sky.fits to a ds9 readable file again and load it:
in src/cheese_script.py:
a.regionextract("mos2S002-bkg_region-sky.fits","mos2_sky_v2.reg")
then:
ds9 rate-mos2S002.fits -scale limits 0 5e-5 -smooth -region load mos2_sky_v2.reg &
Fig.10 Close-up view of the cluster. There are two masks which have to be removed.
Just click on the ellipses and delete them, then save the new region file to newregion.reg. In the top panel of ds9, choose Region --> Save Region. Remember to save in physical coordiantes.
Fig.11 Remeber to choose "physical" in ds9.
If you have difficulty clicking on the ellipses, then on the top panel, choose Edit --> Region .
Fig.12 ds9-->Edit-->Region
Also, when you click on the ellipse, the information is displayed: the number (its position in the file, you may need this number later), the coordinates, the size and the rotation angle.
Fig.13 The informatin displayed when you click on a circle.
If you compare the old and new file, mos2_sky_v2.reg and newregion.reg, you should find 2 masks less. Now we update mos2S002-bkg_region-sky.fits and mos2S002-bkg_region-sky.fits with newregion.reg.
When you remove the ellipses previously, if you notice the number, it should be 1 and 12 that we are going to remove. Now let’s check the file before update:
fdump mos2S002-bkg_region-sky.fits+1 STDOUT "X Y R" 1,12 pagewidth=256 prhead=no clobber=yes X Y R
1 2.4456088E+04 2.4862070E+04 1.6120641E+03
0.0000000E+00 0.0000000E+00 1.6065356E+03
0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
12 2.3949498E+04 2.5638992E+04 5.6302734E+02
0.0000000E+00 0.0000000E+00 5.6109601E+02
0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
The same for mos2S002-bkg_region-det.fits. You see X, Y and R all have four elements, these are for different types of mask (i.e. circle, ellipse, polygon …etc). We are going to set R to 0. We will come back to this file again after update.
In src/cheese_script, we first compare the two files and see which lines have to be removed:
remove_index=a.DeleteMask("pn_sky_v2.reg","newregion.reg")
then we remove the unwanted line:
a.rowchange("mos2S002-bkg_region-sky.fits","R",0,remove_index,[1,2],ccd=mos2)
This line is equivalent to
fpartab 0 mos2S002-bkg_region-sky.fits+1 R row=1 element=1
fpartab 0 mos2S002-bkg_region-sky.fits+1 R row=1 element=2
With fpartab, you change a certain element of a certain row of a certain quantity to a certain value. Here we change the “R” of element 1and 2 or row 1 to 0. Here the “+1” of mos2S002-bkg_region-sky.fits+1 means the first extension, which is REGION. When you do a.rowchange(), mos2S002-bkg_region-det.fits is also updated with coordinate converted using ecoordconv. After the update, we make the cheese image again:
make_mask inimage=mos2S002-obj-im.fits inmask= mos2S002-mask-im.fits outmask= mos2S002-cheese.fits reglist=mos2S002-bkg_region-sky.fits
Now let’s check row 1 and 12 of mos2S002-bkg_region-sky.fits again, you see R is set to 0.
fdump mos2S002-bkg_region-sky.fits+1 STDOUT "X Y R" 1,12 pagewidth=256 prhead=no clobber=yes X Y R
1 2.4456088E+04 2.4862070E+04 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
12 2.3949498E+04 2.5638992E+04 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00
Again the same for mos2S002-bkg_region-det.fits. If you are not familiar with ecoordconv. I suggest that you try converting the sky coordinates to detector coordinate susing this, and cross check with mos2S002-bkg_region-det.fits.
Finally, here are the cheese images we want:
ds9 pnS003-cheese.fits mos1S001-cheese.fits mos2S002-cheese.fits -frame lock image -zoom to fit &
Fig.14 pnS003-cheese.fits, mos1S001-cheese.fits and mos2S002-cheese.fits. Now all three ccds have the same mask.
In the new cheese images, the cluster in the centre is no longer masked.
And these would be used for analysis hereafter.
b. How to add a mask
Now we have got two rows in mos2S002-bkg_region-sky.fits empty. If you want to add a mask, just update these two rows with new values using fpartab or a.ChangeMask() in src/cheese_script.py. For extended sources, cheese always cannot produce a mask large enough.
In case you are not able to modify the values of any row, you really need to “add” a mask, it is a little bit tricky…I am not sure whether there is a direct way to add a row to mos2S002-bkg_region-sky.fits, no matter you are using ftools or python. To the best I know, you combine mos2S002-bkg_region-sky.fits with a new file being the extra mask.
First, we create an empty file called empty.fits using mos2S002-bkg_region-sky.fits, but set all R = 0, or remove unnecessary rows. Then, we input the values of new masks. The original file,mos2S002-bkg_region-sky.fits, has 70 rows, and in this example I need to add two extra masks, so I delete row 1-68 in the following:
cp mos2S002-bkg_region-sky.fits empty.fits
cp mos2S002-bkg_region-det.fits emptydet.fits
fdelrow empty.fits+1 1 68 N Y
Suppose I have to add the following masks (in sky coordinate):
X=17515.893 Y=19017.939 R1=500 R2=500
X=28728.593. Y=19418.211 R1=1006.1019 R2=553.08489 rotation angle=50
I update the above parameters in empty.fits.
fpartab 17515.893 empty.fits+1 X row=1 element=1
fpartab 19017.939 empty.fits+1 Y row=1 element=1
fpartab 500 empty.fits+1 R row=1 element=1
fpartab 500 empty.fits+1 R row=1 element=2
fpartab 0 empty.fits+1 ROTANG row=1 element=1
fpartab 28728.593 empty.fits+1 X row=2 element=1
fpartab 19418.211 empty.fits+1 Y row=2 element=1
fpartab 1006.1019 empty.fits+1 R row=2 element=1
fpartab 553.08489 empty.fits+1 R row=2 element=2
fpartab 50 empty.fits+1 ROTANG row=2 element=1
Check if the update is successful.
fdump empty.fits+1 STDOUT "X Y R" row=1,2 prhead=no showcol=no 1 1.7515893E+04 1.9017939E+04 5.0000000E+02 0.0000000E+00
0.0000000E+00 0.0000000E+00 5.0000000E+02 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
2 2.8728594E+04 1.9418211E+04 1.0061019E+03 5.0000000E+01
0.0000000E+00 0.0000000E+00 5.5308490E+02 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
Now merge empty.fits to mos2S002-bkg_region-sky.fitsfmerge "mos2S002-bkg_region-sky.fits[1] empty.fits[1]" mos2S002-bkg_region-sky.fits - clobber=yes
Now you check the last two rows of mos2S002-bkg_region-sky.fits and you see the new masks are added.
fdump mos2S002-bkg_region-sky.fits+1 STDOUT "X Y R ROTANG" row=71,72 prhead=no showcol=no
The terminal shows the new masks:
21 1.7515893E+04 1.9017939E+04 5.0000000E+02 0.0000000E+00
0.0000000E+00 0.0000000E+00 5.0000000E+02 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
22 2.8728594E+04 1.9418211E+04 1.0061019E+03 5.0000000E+01
0.0000000E+00 0.0000000E+00 5.5308490E+02 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
Not yet done! Remember to run the following to make mos2S002-cheese-new.fitsmake_mask inimage=mos2S002-obj-im.fits inmask=mos2S002-mask-im.fits outmask=mos2S002-cheese-new.fits reglist=mos2S002-bkg_region-sky.fits
(src/cheese_script.py: a,emptyFits(),a.NewMask(),a.Mergefits and a.MakeMask() )
Now check the old and new cheese image in blink mode. You can easily see the new masks.
ds9 mos2S002-cheese-new.fits mos2S002-cheese.fits -lock frame image -blink &
Still we have to update mos2S002-bkg_region-det.fits, but when you run src/cheese_script.py, this file is updated.
Fast way to do it
In src/cheese_script.py, I divide the above process into 5 steps. mos1 is taken as mos1S001, mos2 as mos2S002, pn as pnS003. If these are not correct, open the script and change the lines defining the ccd prefixes at the beginning.
1.Step1 opens the rate image rate-mos2S002.fits loaded with the region file mos2S002-bkg_region-sky.fits, and compare with mos2S002-obj-imsmap.fits, an image with all the sources detected in emllist.fits. If you think mos2S002-bkg_region-sky.fits has all the sources you want, proceed to the next step. If not, run emldetect again with different parameter values.
python src/cheese_script.py 1
Fig.15 rate-mos2S002.fits and mos2S002-obj-imsmap.fits. mos1 and pn are shown together in comparison after running step 1. In this stage, different ccds have different masks. If you are satisfied with the result, go to step 2. If not, run emldetect again with different parameter values.
2. Step2 first updates emllist.fits and reproduce all ***bkg_region-det.fits and ***bkg_region-sky.fits for mos1,mos2 and pn so that all ccds have the same masks. Then, ds9 opens the rate image loaded with the new masks. Delete all the masks you don’t want and save the new region file as newregion.reg in physical coordinates. mos1, mos2 and pn are all shown but you can just use any ccd. Choose the one with not much ccd gaps covering the source you are interested in. You see now all ccds have the same mask.
python src/cheese_script.py 2
Fig.16 rate-mos2S002.fits loaded with updated ***bkg_region-sky.fits converted to ds9 readable format called mos2_sky_v2.reg. The cluster in the centre is our interest. We have to remove the two masks covering it and save the new file as newregion.reg.
3. Step3 returns you the final products you want – updated ***bkg_region-det.fits and ***bkg_region-sky.fits for mos1,mos2 and pn. ds9 opens the cheese images and the rate images loaded with updated mask.
python src/cheese_script.py 3
Fig.17 rate-mos2S002.fits loaded with the updated mask..If you don’t want further changes. This is the final product.
Step4: If you need to enlarge a mask and if there is an empty row in ***bkg_region-sky.fits. Open src/cheese_script.py,go to step 4 and enter the values of the new mask you want. ***bkg_region-det.fits is also updated in this step.
python src/cheese_script.py 4
Step5: If you really need to “add” a mask but not modifying any row, do this step. Open src/cheese_script.py, go to step5 and enter the new mask information.
python src/cheese_script.py 5Reference Websites:
eboxdetect:
http://xmm-tools.cosmos.esa.int/external/sas/current/doc/eboxdetect/
esplinemap:
http://xmm-tools.cosmos.esa.int/external/sas/current/doc/esplinemap/
emldetect:
http://xmm-tools.cosmos.esa.int/external/sas/current/doc/emldetect/
ftools:
https://heasarc.gsfc.nasa.gov/lheasoft/ftools/futils.html
Fig.1 an image extracted from mos2S002-clean.fits in [0.4 – 2.3] keV. This image shows the photons in this energy range.
exposure map
Fig.2 an exposure map in [0.4 – 2.3] keV. This has been corrected for vignetting effect. Otherwise, all pixels have the same value.
mask
Fig.3 A mask indicating ccd gaps and bad pixels. All pixels have either a value of 0 or 1. Point source searches only perform on pixels with a value = 1.
2. run eboxdetect for the first time for local detection to produce a file called eboxlist_l.fits. This file contains detected point sources information.
Fig.4 mos2S002-obj-imbkg.fits, A background map produced from esplinemap.
4. Run eboxdetect for the second time for map detection using the background map produced from the previous step(mos2S002-obj-imbkg.fits). A file called eboxlist_m.fits is produced, which has more sources (336) than eboxlist_l.fits
Fig.5 pnS003-obj-imsmap.fits,mos1S001-obj-imsmap.fits and mos2S002-obj-imsmap.fits. The images display the sources detected in emllist.fits.
Now you can test by running emldetect again, but change mlmin to a larger value, e.g. 150, and open mos2S002-obj-imsmap.fits. You find there are a lot less sources.
6&7. Now we have finalized the sources in emllist.fits. We have to make further cuts to have the final list. Run region twice to create two fits files which contain the size and coordinates of the detected point sources satisfying selection criteria in sky and detector coordinates. The files are called mos2S002-bkg_region-sky.fits (for image data, this file needs further processing) and mos2S002-bkg_region-det.fits (for spectroscopic data).
Remember at the beginning you input some parameters when calling cheese? Here are when these parameters come into play!
Fig.6 left: rate-mos2S002.fits. right: mos2S002-obj-imsmap.fits, an image showing all sources in emllist.fits.
If you think it is not good, then do region again, lowering the threshold of DET_ML.
Let’s check further all the ccds.
Fig.7 top left: rate-mos1S001.fits,top right: rate-mos2S002.fits, bottom left: rate-pnS003.fits
pn finds a lot more point sources! I believe the reason is pn detects photons a few times higher than mos1 and mos2, resulting in a higher detection likelihood. Now let’s load the rate images of mos2 but with the pn region file loaded, and compare with the one you just loaded:
ds9 rate-mos2S002.fits -scale limits 0 5e-5 -smooth -region load mos2_sky.reg rate-mos2S002.fits -scale limits 0 5e-5 -smooth -region load pn_sky.reg -zoom to fit -frame prev -zoom to fit &
Fig.8 Close-up of rate-mos2S002.fits. The left one is loaded with mos2’s mask. The right one is loaded with pn’s mask. Green ellipses are common masks to both, cyan only found in mos2 and red only found in pn.
In the above figure, you see in some red ellipses, there is a faint source, but just too dim to pass the selection criteria. The opposite also happens. There are sources found in mos2 but not pn. If you want all ccds having the same mask, we have to do region again but choose the summary band, i.e. (ID_INST ==0 )&&(ID_BAND == 0). But if you open emllist.fits, you find for ID_BAND=0, there are a lot of FLUX with “INDEF”.
Let’s take a look:
Fig.9 pnS003-cheese.fits mos1S001-cheese.fits and mos2S002-cheese.fits. After update, all ccds have the same masks.
8. remove or add a mask
a. How to remove a mask
From the above, we see the cluster in the centre is also masked. This is not what we want, we now have to remove it. The easiest way is to open mos2S002-bkg_region-sky.fits and mos2S002-bkg_region-sky.fits with fv, and then manually change the radii to 0.
If you don’t want to do it manually, let’s do it with other commands in ftools. First, we need to decide which mask we have to remove, we convert mos2S002-bkg_region-sky.fits to a ds9 readable file again and load it:
in src/cheese_script.py:
Fig.10 Close-up view of the cluster. There are two masks which have to be removed.
Just click on the ellipses and delete them, then save the new region file to newregion.reg. In the top panel of ds9, choose Region --> Save Region. Remember to save in physical coordiantes.
Fig.11 Remeber to choose "physical" in ds9.
If you have difficulty clicking on the ellipses, then on the top panel, choose Edit --> Region .
Fig.12 ds9-->Edit-->Region
Also, when you click on the ellipse, the information is displayed: the number (its position in the file, you may need this number later), the coordinates, the size and the rotation angle.
Fig.13 The informatin displayed when you click on a circle.
If you compare the old and new file, mos2_sky_v2.reg and newregion.reg, you should find 2 masks less. Now we update mos2S002-bkg_region-sky.fits and mos2S002-bkg_region-sky.fits with newregion.reg.
When you remove the ellipses previously, if you notice the number, it should be 1 and 12 that we are going to remove. Now let’s check the file before update:
Fig.14 pnS003-cheese.fits, mos1S001-cheese.fits and mos2S002-cheese.fits. Now all three ccds have the same mask.
In the new cheese images, the cluster in the centre is no longer masked.
And these would be used for analysis hereafter.
b. How to add a mask
Now we have got two rows in mos2S002-bkg_region-sky.fits empty. If you want to add a mask, just update these two rows with new values using fpartab or a.ChangeMask() in src/cheese_script.py. For extended sources, cheese always cannot produce a mask large enough.
In case you are not able to modify the values of any row, you really need to “add” a mask, it is a little bit tricky…I am not sure whether there is a direct way to add a row to mos2S002-bkg_region-sky.fits, no matter you are using ftools or python. To the best I know, you combine mos2S002-bkg_region-sky.fits with a new file being the extra mask.
First, we create an empty file called empty.fits using mos2S002-bkg_region-sky.fits, but set all R = 0, or remove unnecessary rows. Then, we input the values of new masks. The original file,mos2S002-bkg_region-sky.fits, has 70 rows, and in this example I need to add two extra masks, so I delete row 1-68 in the following:
Fig.15 rate-mos2S002.fits and mos2S002-obj-imsmap.fits. mos1 and pn are shown together in comparison after running step 1. In this stage, different ccds have different masks. If you are satisfied with the result, go to step 2. If not, run emldetect again with different parameter values.
2. Step2 first updates emllist.fits and reproduce all ***bkg_region-det.fits and ***bkg_region-sky.fits for mos1,mos2 and pn so that all ccds have the same masks. Then, ds9 opens the rate image loaded with the new masks. Delete all the masks you don’t want and save the new region file as newregion.reg in physical coordinates. mos1, mos2 and pn are all shown but you can just use any ccd. Choose the one with not much ccd gaps covering the source you are interested in. You see now all ccds have the same mask.
Fig.16 rate-mos2S002.fits loaded with updated ***bkg_region-sky.fits converted to ds9 readable format called mos2_sky_v2.reg. The cluster in the centre is our interest. We have to remove the two masks covering it and save the new file as newregion.reg.
3. Step3 returns you the final products you want – updated ***bkg_region-det.fits and ***bkg_region-sky.fits for mos1,mos2 and pn. ds9 opens the cheese images and the rate images loaded with updated mask.
Fig.17 rate-mos2S002.fits loaded with the updated mask..If you don’t want further changes. This is the final product.
Step4: If you need to enlarge a mask and if there is an empty row in ***bkg_region-sky.fits. Open src/cheese_script.py,go to step 4 and enter the values of the new mask you want. ***bkg_region-det.fits is also updated in this step.